Heat trace area extraction apparatus, heat trace area extraction method and program

ABSTRACT

A heat trace region extraction device includes an information output unit  181  that outputs information indicating a heat trace region which is a region of a trace of heat, extracted based on an actual object image which is an image of an actual object, obtained by photographing a certain range with an actual object camera for photographing the actual object and a thermal image which is an image of the heat emitted by the actual object, obtained by photographing a certain range with a thermal camera for photographing the heat emitted by the actual object. 
     The information output unit  181  displays information indicating the heat trace region on a transmissive display so that the heat trace region existing in the certain range and information indicating the heat trace region displayed on the transmissive display are aligned with each other.

TECHNICAL FIELD

The present invention relates to a heat trace region extraction device,a heat trace region extraction method and a program.

BACKGROUND ART

With the epidemic of the new coronavirus infection (COVID-19), variousmeasures have been taken to prevent infection, one of which isdisinfection or sterilization. If the mucous membrane of the mouth,nose, or eyes is touched with a finger to which this virus is attached,there will be infection with the virus. Therefore, it is recommended todisinfect not only the fingers but also personal items that an infectedperson may have touched. Therefore, in restaurants and sports gyms usedby many people, measures are taken such as regularly wiping desks,doors, training equipment, and the like with alcohol disinfectant or thelike.

It is difficult to continue visually checking which portions haveactually been touched. In stores, measures such as disinfecting allplaces that may have been touched by people are often taken at regularintervals. However, the task of disinfecting all places that may havebeen touched by people on a regular basis involves a large amount oflabor. Therefore, a method for improving the efficiency of disinfectionusing a drone has been proposed (for example, NPL 1).

On the other hand, it is also conceivable to use a surveillance cameraor the like to detect the place where persons are present and disinfectonly that place. There has been much research on human detection usingvideo, and in recent years it has become possible to identify the placewhere a person is present from a video with considerably high accuracy.If it is possible to disinfect only the place where a person was presentusing such a technique, it is considered that the labor for disinfectioncan be reduced. For example, it is not necessary to disinfect the regionaround a place when it has not been used by people at all. On the otherhand, if it is known that a plurality of persons have used a place, itshould be possible to disinfect the place at an early stage.Disinfection at regular time intervals cannot prevent infectionsmediated by objects, that is, infections caused by an infected persontouching an object and another person touching the object within aninterval. However, if there can be disinfection flexibly according tohuman use, it is thought that such spread of infection can be furtherreduced. In addition, since such a method can prevent unnecessarydisinfection, the effect of reducing the amount of disinfectant solutionused can be expected. In other words, if disinfection can be performedaccording to the use of objects by a person detected by surveillancecameras, labor will be reduced, the spread of infection can beprevented, and disinfectant solution can be saved as compared with thecase of disinfecting all objects that may have been used at regular timeintervals.

If it is possible to disinfect only the places that people actuallytouched in a pinpoint manner, instead of disinfecting everything aroundthe place where people were, it would be possible to further reducelabor and disinfectant solution. However, it is difficult to identifywhether a person who was actually at a place touched an object by themethod using a visible video captured by a surveillance camera or thelike. For example, supposing that a camera is installed downward fromthe ceiling and a table is being imaged, when a hand is extended ontothe table, it may be difficult to determine whether the hand is touchingthe desk from the video taken by the installed camera. Therefore, amethod for detecting contact with an object using a shadow has beenproposed (NPL 2).

CITATION LIST Non Patent Literature

-   [NPL 1] “Efficiently disinfecting a stadium with a drone! Invented    by US startup with the new Corona,” [online], Internet,    <URL:https://techable.jp/archives/124749>-   [NPL 2] Tatsuo Yoshida, Hon Yaokai, Seiichi Uchida, Contact    detection using image recognition, Electrical Society, Kyushu Branch    Joint Conference 2011.

SUMMARY OF INVENTION Technical Problem

However, the method of NPL 2 requires a strong light source such as aprojector. In addition, it is assumed that the recognition accuracy issignificantly affected by the positional relationship between the cameraand the light source. A strong light source cannot be installed freelyin many environments, and it is considered that it is not suitable forthe purpose of detecting and presenting a place touched by a person invarious places to support disinfection.

The present invention has been made in view of the above-mentionedproblems, and an object thereof is to present information such as placestouched by persons such that this can be easily grasped by a user.

Solution to Problem

In order to solve the above-mentioned problem, a heat trace regionextraction device includes an information output unit that outputsinformation indicating a heat trace region which is a region of a traceof heat, extracted based on an actual object image which is an image ofan actual object, obtained by photographing a certain range with anactual object camera for photographing the actual object and a thermalimage which is an image of the heat emitted by the actual object,obtained by photographing a certain range with a thermal camera forphotographing the heat emitted by the actual object, wherein theinformation output unit displays information indicating the heat traceregion on a transmissive display so that the heat trace region existingin the certain range and information indicating the heat trace regiondisplayed on the transmissive display are aligned with each other.

A heat trace region extraction device may include an information outputunit that outputs information indicating a heat trace region which is aregion of a trace of heat, extracted based on an actual object imagewhich is an image of an actual object, obtained by photographing acertain range with an actual object camera for photographing the actualobject and a thermal image which is an image of the heat emitted by theactual object, obtained by photographing a certain range with a thermalcamera for photographing the heat emitted by the actual object, whereinthe information output unit displays an image or a video in whichinformation indicating the heat trace region is superimposed on an imageor a video of the certain range on a display so that the actual objectexisting in the certain range and an image of the actual object includedon the image or the video displayed on the display are aligned with eachother.

An heat trace region extraction device may include an information outputunit that outputs information indicating a heat trace region which is aregion of a trace of heat, extracted based on an actual object imagewhich is an image of an actual object, obtained by photographing acertain range with an actual object camera for photographing the actualobject and a thermal image which is an image of the heat emitted by theactual object, obtained by photographing a certain range with a thermalcamera for photographing the heat emitted by the actual object, whereinthe information output unit outputs the information indicating the heattrace region via sound.

A heat trace region extraction device may include an information outputunit that outputs information indicating a heat trace region which is aregion of a trace of heat, extracted based on an actual object imagewhich is an image of an actual object, obtained by photographing acertain range with an actual object camera for photographing the actualobject and a thermal image which is an image of the heat emitted by theactual object, obtained by photographing a certain range with a thermalcamera for photographing the heat emitted by the actual object, whereinthe information output unit displays the information indicating the heattrace region via text.

A heat trace region extraction device may include an information outputunit that outputs information indicating a region other than a region ofa trace of heat, extracted based on an actual object image which is animage of an actual object, obtained by photographing a certain rangewith an actual object camera for photographing the actual object and athermal image which is an image of the heat emitted by the actualobject, obtained by photographing a certain range with a thermal camerafor photographing the heat emitted by the actual object.

A heat trace region extraction device may include an information outputunit that outputs information indicating a heat trace region which is aregion of a trace of heat, extracted based on an actual object imagewhich is an image of an actual object, obtained by photographing acertain range with an actual object camera for photographing the actualobject and a thermal image which is an image of the heat emitted by theactual object, obtained by photographing a certain range with a thermalcamera for photographing the heat emitted by the actual object, whereinthe information output unit outputs different information depending onthe number of times the heat trace region was extracted as informationindicating the heat trace region.

Advantageous Effects of Invention

According to the present invention, it is possible to presentinformation such as a place touched by a person so that the user caneasily grasp it.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a visible image and a thermal image ofthe same place taken at the same time.

FIG. 2 is a diagram showing an example of an image obtained bybackground subtraction.

FIG. 3 is a diagram showing a hardware configuration example of a heattrace region extraction device 10 according to the first to fifthembodiments.

FIG. 4 is a diagram showing a functional configuration example of theheat trace region extraction device 10 in the first embodiment.

FIG. 5 is a flowchart for explaining an example of a processingprocedure executed by the heat trace region extraction device 10 in thefirst embodiment.

FIG. 6 is a schematic diagram showing an output example of informationindicating a heat trace region.

FIG. 7 is a diagram showing a functional configuration example of a heattrace region extraction system 1 and the heat trace region extractiondevice 10 in a modified example of the first embodiment.

FIG. 8 is a flowchart for explaining an example of a processingprocedure executed by a modified example of the first embodiment.

FIG. 9 is a diagram showing a functional configuration example of theheat trace region extraction system 1 and the heat trace regionextraction device 10 in the second embodiment.

FIG. 10 is a flowchart for explaining an example of a processingprocedure executed by the heat trace region extraction device 10 in thesecond embodiment.

FIG. 11 is a diagram for explaining the second embodiment.

FIG. 12 is a diagram for explaining the second embodiment.

FIG. 13 is a diagram showing a functional configuration example of theheat trace region extraction system 1 and the heat trace regionextraction device 10 in the third embodiment.

FIG. 14 is a flowchart for explaining an example of a processingprocedure executed by the heat trace region extraction device 10 in thethird embodiment.

FIG. 15 is a diagram for explaining the third embodiment.

FIG. 16 is a diagram for explaining the third embodiment.

FIG. 17 is a diagram for explaining the third embodiment.

FIG. 18 is a diagram for explaining the third embodiment.

FIG. 19 is a diagram for explaining the third embodiment.

FIG. 20 is a diagram showing a functional configuration example of theheat trace region extraction system 1 and the heat trace regionextraction device 10 in the fourth embodiment.

FIG. 21 is a flowchart for explaining an example of a processingprocedure executed by the heat trace region extraction device 10 in thefourth embodiment.

FIG. 22 is a diagram for explaining the fifth embodiment.

DESCRIPTION OF EMBODIMENTS [Overview]

In the first embodiment, a device, a method, and a program for detectinga place touched by a person using a thermal image are disclosed with theaim to help sterilize or disinfect bacteria or viruses. Since a personwho is a homeothermic animal has heat on his/her limbs, when the persontouches an object, the heat remains in the place touched by the personfor a certain period of time. For example, a method of abusing the heattrace, which is a trace of human touch identified by the heat remainingin the place touched by the person, to decrypt the passcode of asmartphone has been reported (“Yomna Abdelrahman, Mohamed Khamis, StefanSchneegass, and Florian Alt. 2017. Stay Cool! Understanding ThermalAttacks on Mobile-based User Authentication. In Proceedings of the 2017CHI Conference on Human Factors in Computing Systems (CHI '17), pp.3751.3763, 2017”).

Heat traces remain not only on the screens of smartphones, but also onvarious places such as desks and walls. Therefore, if the heat trace isidentified based on the video (thermal image) of a thermal camera, theplace touched by a person indoors or the like can be pinpointed. Athermal image is an image (that is, an image obtained by photographingrays of heat radiated by an actual object) of heat emitted by an actualobject, obtained by photographing a certain range with a thermal camerafor photographing the heat emitted by the actual object (that is, therays of heat radiated by the actual object and the electromagnetic waveshaving the wavelength in the far-infrared region). More specifically,the thermal image is an image of a temperature distribution by detectingthe infrared radiation energy emitted from an object and converting itinto an apparent temperature. In other words, a thermal image is not animage of the infrared rays reflected by the actual object.

The heat trace region can be extracted by background subtraction withthe thermal image before being touched by a person as the background.However, since the human body itself is hot, this method extracts thehuman body region as well as the heat trace. Therefore, in the firstembodiment, a visible image is acquired at the same time as the thermalimage, and the heat trace region is extracted by comparing the thermalimage with the visible image.

Specifically, background subtraction is performed for each of thevisible image and the thermal image, and the heat trace region isextracted by the difference in the result of the background subtraction.Since heat traces cannot be observed with a visible image (that is, withthe naked eye), they cannot be extracted even if background subtractionis performed on the visible image with the visible image before beingtouched by a person as the background. On the other hand, when there isa person on the spot, the region of the person is extracted byperforming background subtraction with the visible image taken in theabsence of the person as the background. That is, when the regionextracted by background subtraction in the thermal image is similarlyextracted in the visible image, it can be seen that the region is not aheat trace. On the other hand, the region extracted in the thermal imageby background subtraction and not extracted in the visible image islikely to be a heat trace. In the first embodiment, the heat traceregion extracted by such a method is visualized, and the place touchedby a person is transmitted to the user. For simultaneous acquisition ofthermal and visible images, a device such as a sensor node equipped witha visible light camera and a thermal camera may be used (“YoshinariShirai, Yasue Kishino, Takayuki Suyama, Shin Mizutani: PASNIC: a thermalbased privacy-aware sensor node for image capturing, UbiComp/ISWC '19Adjunct, pp. 202-205, 2019”).

First Embodiment

The object of the first embodiment will be described with reference toFIGS. 1 and 2 . FIG. 1 is a schematic diagram of a visible image and athermal image of the same place taken at the same place. FIG. 1 shows aschematic diagram of an image of a hand touching a door with a handletaken simultaneously with a visible light camera and a thermal camera.FIG. 1(a) and FIG. 1 (a′) are a visible image and a thermal image attime t1 (before the hand touches the door), respectively. FIG. 1(b) andFIG. 1 (b′) are a visible image and a thermal image at time t2 (a statein which the hand is touching the door), respectively. FIG. 1(c) andFIG. 1 (c′) are a visible image and a thermal image at time t3 (afterthe hand touches the door). When a person touches the door, thetemperature of the touched place rises as shown in FIG. 1 (c′).

FIG. 2 is a diagram showing an example of an image obtained bybackground subtraction. FIG. 2 shows a difference image which is thebackground subtraction between the image at time t2 and the image attime t3 when the image at time t1 in FIG. 1 is used as the backgroundimage. At time t2, the arm portion is extracted as a difference regionwhere there is a difference from the background image in both thevisible image and the thermal image. In contrast, at time t3, theportion touching the door is extracted as a difference region only inthe thermal image.

Considering disinfecting the region actually touched by the hand, thedifference region extracted by background subtraction with respect tothe thermal image (background thermal image) at time t1 of the thermalimage at time t3 may be disinfected. On the other hand, the differenceregion extracted by background subtraction with respect to the thermalimage (background thermal image) at time t1 of the thermal image at timet2 includes a portion not touching the door. In the first place, thedifference region extracted by the background subtraction of the thermalimage at time t2 with respect to the thermal image (background thermalimage) at time t1 is the region where the human body existed, and is notthe region of heat trace left by actually touching the door. From theviewpoint of disinfection, it is sufficient to identify the differenceregion extracted by the background subtraction at time t3, and thedifference region extracted by the background subtraction at time t2 isunnecessary.

Therefore, in the first embodiment, when a similar difference region isextracted by background subtraction even in a visible image, it isdetermined that the difference region is not a heat trace region. Attime t2, since the shape of the arm is also extracted in the visibleimage, it is determined that the difference region extracted in thethermal image is not a heat trace region (that is, a portion touched bya person). On the other hand, at time t3, it is determined that theregion extracted by the background subtraction of the thermal image isnot extracted in the visible image, so that the difference regionextracted in the thermal image is the heat trace region (that is, theportion touched by a person). If a system presents informationindicating the heat trace region extracted based on such adetermination, the user who sees the information can efficientlydisinfect the portion touched by a person.

Hereinafter, the heat trace region extraction device 10 that realizesthe above will be specifically described. FIG. 3 is a diagram showing ahardware configuration example of the heat trace region extractiondevice 10 in the first embodiment, a modified example described later ofthe first embodiment, and the second to fifth embodiments. The heattrace region extraction device 10 of FIG. 3 has a drive device 100, anauxiliary storage device 102, a memory device 103, a CPU 104, aninterface device 105, and the like, which are connected to each other bya bus B.

The program that realizes the processing in the heat trace regionextraction device 10 is provided by a recording medium 101 such as aCD-ROM. When the recording medium 101 storing the program is set in thedrive device 100, the program is installed in the auxiliary storagedevice 102 from the recording medium 101 via the drive device 100.However, the program does not necessarily have to be installed from therecording medium 101, and may be downloaded from another computer viathe network. The auxiliary storage device 102 stores the installedprogram and also stores necessary files, data, and the like.

The memory device 103 reads and stores the program from the auxiliarystorage device 102 when an instruction for starting the program isissued. The CPU 104 executes the function related to the heat traceregion extraction device 10 according to the program stored in thememory device 103. The interface device 105 is used as an interface forconnecting to a network.

FIG. 4 is a diagram showing a functional configuration example of theheat trace region extraction device 10 in the first embodiment. In FIG.4 , the heat trace region extraction device 10 includes a visible imageacquisition unit 11, a background visible image generation unit 12, adifference visible image generation unit 13, a thermal image acquisitionunit 14, a background thermal image generation unit 15, a differencethermal image generation unit 16, a heat trace region extraction unit17, a heat trace region output unit 18, and the like. Each of theseunits is realized by one or more programs installed in the heat traceregion extraction device 10 causing the CPU 104 to execute processing.

As shown in FIG. 4 , the heat trace region extraction device 10 isconnected to each of a visible light camera 21 and a thermal camera 22so that images can be input from the cameras. The visible light camera21 and the thermal camera 22 are installed so as to be able tophotograph the same certain place (the same certain range). That is, thefirst embodiment is based on the assumption that the photographingregion of the visible light camera 21 and the photographing region ofthe thermal camera 22 match each other on a pixel-by-pixel basis. If thephotographed portions of the visible light camera 21 and the thermalcamera 22 do not match, calibration may be performed in advance so thatthe correspondence between the pixels of the visible image and thethermal image can be grasped.

FIG. 5 is a flowchart for explaining an example of a processingprocedure executed by the heat trace region extraction device 10.

In step S101, the visible image acquisition unit 11 acquires the visibleimage photographed by the visible light camera 21, input from thevisible light camera 21, and the thermal image acquisition unit 14acquires the thermal image photographed by the thermal camera 22, inputfrom the thermal camera 22. In step S101, the acquisition of the visibleimage by the visible image acquisition unit 11 and the acquisition ofthe thermal image by the thermal image acquisition unit 14 may or maynot be performed at the same time. If not at the same time, a portion ofthe frame of the camera with the faster frame rate may be ignoredaccording to the camera with the slower frame rate. Further, there is noproblem even if still images are alternately acquired from the visiblelight camera 21 and the thermal camera 22 and the acquired images areregarded as being acquired at the same time as long as the frame rate ishigh to some extent. The visible image acquisition unit 11 transmits theacquired visible image to the background visible image generation unit12, and the thermal image acquisition unit 14 transmits the acquiredthermal image to the background thermal image generation unit 15.

Subsequently, the background visible image generation unit 12 stores thevisible image transmitted from the visible image acquisition unit 11 inthe auxiliary storage device 102, and the background thermal imagegeneration unit 15 stores the thermal image transmitted from the thermalimage acquisition unit 14 in the auxiliary storage device 102 (S102).

Steps S101 and S102 are repeated until a predetermined time T1 elapsesfrom the start of execution. The predetermined time T1 may be a periodin which one or more visible images and one or more thermal images areaccumulated in the auxiliary storage device 102.

When the predetermined time T1 has elapsed from the start of executionof the first step S101 (Yes in S103), the processing proceeds to stepS104. In step S104, the background visible image generation unit 12generates a background image (hereinafter, referred to as “backgroundvisible image”) in the photographing range based on the visible imagegroup stored in the auxiliary storage device 102 at the predeterminedtime T1. In step S104, the background thermal image generation unit 15generates a background image (hereinafter, referred to as “backgroundthermal image”) in the photographing range based on the thermal imagegroup stored in the auxiliary storage device 102 in the predeterminedtime T1.

For example, when there are a plurality of photographed images includedin a visible image group or a thermal image group (hereinafter, when thevisible image group and the thermal image group are not distinguished,simply referred to as “photographed image group”), a background image(background visible image and background thermal image) may be generatedfor each image group using the average value or the median value of thepixel values (RGB) of each photographed image group as the pixel valuesof each pixel. When there is one photographed image included in thephotographed image group, the photographed image is regarded as abackground image. By doing so, it is possible to remove a person who haspassed temporarily from the background image, and it is possible togenerate a background image which is an image composed of only objectsthat remain in the photographing range for a long time. Many studieshave been conducted on a method of dynamically creating a backgroundfrom images photographed for a certain period of time, and the method ofgenerating a background image in the first embodiment is not limited toa predetermined method.

The predetermined time T1 is a time interval corresponding to the timet1 in FIG. 1 , and includes the time t1.

When the background visible image and the background thermal image aregenerated, step S105 and subsequent steps are executed. It should benoted that steps S101 to S104 and step S105 do not have to be executedsynchronously. For example, step S105 and the subsequent steps may bestarted in response to an instruction different from the executioninstruction of steps S101 to S104.

In step S105, the visible image acquisition unit 11 and the thermalimage acquisition unit 14 wait for the elapse of a predetermined timeT2. The predetermined time T2 is, for example, the elapsed time from thetime t2 to the time t3 in FIG. 2 .

When the predetermined time T2 elapses (Yes in S105), the visible imageacquisition unit 11 acquires a visible image (hereinafter referred to as“target visible image”) input from the visible light camera 21, and thethermal image acquisition unit 14 acquires a thermal image (hereinafter,referred to as “target thermal image”) input from the thermal camera 22(S106). It is desirable that the target visible image and the targetthermal image are images taken at the same time (or almost at the sametime).

In the subsequent step S107, the difference visible image generationunit 13 compares the background visible image generated by thebackground visible image generation unit 12 with the target visibleimage by the background subtraction method, and extracts a differenceregion (a region different from the background visible image) withrespect to the background visible image from the target visible image tothereby generate a difference image showing the difference (hereinafterreferred to as “difference visible image”). The difference thermal imagegeneration unit 16 compares the background thermal image generated bythe background thermal image generation unit 15 with the target thermalimage according to the background subtraction method, and extracts adifference region (a region different from the background thermal image)with respect to the background thermal image to generate a differenceimage (hereinafter, referred to as “difference thermal image”)indicating the difference. If the difference in pixel values withrespect to the background image is equal to or greater than a certainthreshold value, the pixels are different from the background, and ifnot, the pixels are the same as the background. Then, for example, abinary image in which the pixel value of the pixel different from thebackground is 1 and the pixel value of the same pixel as the backgroundis 0 is generated as the difference images (difference visible image anddifference thermal image). Further, the difference images are sent tothe heat trace region extraction unit 17.

Subsequently, the heat trace region extraction unit 17 compares thedifference visible image with the difference thermal image to extractthe heat trace region in the photographing range (S108).

When extracting a region dissimilar to the difference visible region ofthe difference thermal image, the similarity determination of thedifference region of each difference image may be used. For example, theheat trace region extraction unit 17 first labels (extracts theconnecting region) each binary image which is a difference visible imageor a difference thermal image. Next, the heat trace region extractionunit 17 compares the degrees of overlap of one or more differenceregions (hereinafter, referred to as “difference visible region”)obtained by labeling the difference visible image for each of one ormore difference regions (hereinafter referred to as “difference thermalregion”) obtained by labeling the difference thermal images.Specifically, the heat trace region extraction unit 17 counts whetherthe difference regions to be compared match each other on apixel-by-pixel basis, and if the match rate is less than a certainthreshold value, it is determined that the two difference regionscompared are dissimilar. The heat trace region extraction unit 17extracts a difference thermal region that is dissimilar to any of thedifference visible regions as a heat trace region.

The heat trace region extraction unit 17 transmits informationindicating the heat trace region and a background visible image to theheat trace region output unit 18. At this time, the heat trace regionextraction unit 17 may generate a binary image in which the heat traceregion portion is white and the rest is black, and transmit informationindicating whether the binary image is the heat trace region to the heattrace region output unit 18 as information indicating the heat traceregion. It should be noted that the determination of the similarity ofregions is actively performed in the research of pattern matching andthe like and is not limited to a predetermined method.

Subsequently, the heat trace region output unit 18 outputs informationindicating the heat trace region so that the user can confirm it (S109).Here, the user is a person to be notified of the information indicatingthe heat trace region. For example, the heat trace region output unit 18may output an image obtained by combining white pixels of a binaryimage, which is an example of information indicating a heat traceregion, on a background visible image. The output form is not limited toa predetermined form. For example, the display on a display device, thestorage in the auxiliary storage device 102, the transmission to a userterminal via a network, and the like may be performed.

Subsequently to step S109, steps S105 and subsequent steps are repeated.Alternatively, step S109 may be executed after steps S105 to S108 arerepeated a plurality of times. In this case, the heat trace regionsextracted in the plurality of times can be collectively output.

FIG. 6 is a schematic diagram showing an example of an image output bythe heat trace region output unit 18. In FIG. 6 , the portion touched bya person hand is painted black (however, black is a color forconvenience, and the actual color may be a different color such aswhite). The user can recognize the portion as a heat trace region.

The heat trace region output unit 18 may project a binary image showingthe heat trace region to the photographing range in the environmentafter performing alignment appropriately using a projector or the like.In this case, the heat trace region image is projected on the heat traceportion in the environment, and the portion touched by a person can bedirectly transmitted to each user in the environment. Using such amethod, the user can know the heat trace region, and thus, the user canbe urged to take an action such as avoiding touching the heat traceregion.

Further, steps S101 to S103 may be executed in parallel with step S105and subsequent steps. In this case, the background visible image and thebackground thermal image are updated periodically. Therefore, it can beexpected that the resistance to the change in the background with thepassage of time will be improved.

In the above, it is assumed that the camera is fixed to an indoor placeand heat traces left on a wall or a desk where the place is relativelyfixed are extracted. However, the first embodiment can be applied to amoving object as long as the position in the image of an object to betouched by a person can be identified. For example, if a QR code(registered trademark) for identifying the position is attached to thefour corners of the seat surface of a chair and the QR code (registeredtrademark) can be used as a clue to estimate the position on the seatsurface, even if the chair moves, the heat traces left on the seatsurface can be estimated with the seat surface as a background, and theheat traces can be displayed on the seat surface on the image. Varioustechniques for estimating the position of an object in an image havebeen proposed, and the technique is not limited to the use of a QR code(registered trademark).

As described above, according to the first embodiment, it is possible toimprove the detection accuracy of the place touched by a person. As aresult, for example, it is possible to efficiently sterilize anddisinfect a place where viruses such as a new coronavirus or bacteria(hereinafter, referred to as “viruses” for convenience) may be attached.

In the first embodiment, the difference visible image is an example of afirst difference image. The difference thermal image is an example of asecond difference image. The heat trace region extraction unit 17 is anexample of an extraction unit.

Modified Example of First Embodiment

In the first embodiment, in order to explain the difference from thethermal image in an easy-to-understand manner, an example of using avisible image acquired by a visible light camera as an image of anactual object such as a person or an object existing in a photographingrange has been described. However, naturally, an image of an actualobject may be taken in any wavelength band, and this embodiment will bedescribed as a modified example of the first embodiment.

As shown in FIG. 7 , the heat trace region extraction system 1 of themodified example of the first embodiment includes a heat trace regionextraction device 10, an actual object camera 210, and a thermal camera22. As shown in FIG. 7 , the heat trace region extraction device 10 ofthe modified example of the first embodiment includes, for example, anactual object image acquisition unit 110, a background actual objectimage generation unit 120, a difference actual object image generationunit 130, a thermal image acquisition unit 14, a background thermalimage generation unit 15, a difference thermal image generation unit 16,a heat trace region extraction unit 17, and a heat trace region outputunit 18. The actual object camera 210 and the thermal camera 22 areconnected to the heat trace region extraction device 10 of the modifiedexample of the first embodiment, and the images taken by the actualobject camera 210 and the thermal camera 22 are input to the heat traceregion extraction device 10.

The heat trace region extraction method of the modified example of thefirst embodiment is realized by each unit of the heat trace regionextraction device executing the processing of steps S101 to S109 shownin FIG. 8 and below. Hereinafter, with respect to the modified exampleof the first embodiment, the same portions as those of the firstembodiment will be appropriately omitted, and the portions differentfrom the first embodiment will be mainly described.

Actual Object Camera 210

The actual object camera 210 is a camera for photographing an actualobject. The actual object camera 210 is the same as the visible lightcamera 21 of the first embodiment except that the wavelength band of theelectromagnetic waves obtained by the actual object camera 210 is notlimited to the wavelength band of the visible light. The actual objectcamera 210 photographs an image of an actual object in a certainphotographing range, more specifically, an actual object image which isan image composed of an image of an actual object existing in thephotographing range as viewed from the actual object camera 210 side.The actual object image photographed by the actual object camera 210 isinput to the actual object image acquisition unit 110.

The wavelength band of the electromagnetic waves photographed by theactual object camera 210 may be any wavelength band in which the actualobject can be photographed. A specific example of the wavelength band ofthe electromagnetic waves obtained by the actual object camera 210 willbe described in the fourth embodiment. However, when the wavelength bandof the electromagnetic waves obtained by the actual object camera 210does not include the wavelength band of the electromagnetic wavesexisting in the photographing range due to illumination or the like, asindicated by a broken line in FIG. 7 , the heat trace region extractionsystem 1 of the modified example of the first embodiment may be providedwith an irradiator 41 so that the photographing range may be irradiatedwith the electromagnetic waves having the wavelength band obtained bythe actual object camera 210 using the irradiator 41 that emits anelectromagnetic waves having the wavelength band obtained by the actualobject camera 210. An actual object is one that has an entity. Examplesof an actual object existing in the photographing range are an objectexisting as a background in the photographing range, a person in thephotographing range, and a person entering the photographing range.

Actual Object Image Acquisition Unit 110

The actual object image acquisition unit 110 is the same as the visibleimage acquisition unit 11 of the first embodiment, except that itacquires and outputs an actual object image instead of the visibleimage. That is, the actual object image acquisition unit 110 acquiresthe actual object image photographed by the actual object camera 210(S101) and outputs the acquired actual object image. The actual objectimage acquired by the actual object image acquisition unit 110 is inputto the background actual object image generation unit 120 and/or thedifference actual object image generation unit 130.

Background Actual Object Image Generation Unit 120

The background actual object image generation unit 120 is the same asthe background visible image generation unit 12 of the first embodiment,except that it processes an actual object image instead of the visibleimage and that it generates and outputs a background actual object imageinstead of the background visible image. That is, the background actualobject image generation unit 120 generates a background actual objectimage which is an actual object image of the background in thephotographing range based on the actual object image acquired by theactual object image acquisition unit 110 (S104) and outputs thegenerated background actual object image. The actual object image of thebackground in the photographing range is an image of an object existingas a background in the photographing range, and more specifically, animage composed of images of an object existing as a background in thephotographing range, viewed from the actual object camera 210 side. Thegenerated background actual object image is input to the differenceactual object image generation unit 130.

<Difference Actual Object Image Generation Unit 130>

The difference actual object image generation unit 130 is the same asthe difference visible image generation unit 13 of the first embodimentexcept that it performs processing on an actual object image instead ofa visible image, that it performs processing on a background actualobject image instead of a background visible image, and that itgenerates a difference actual object image instead of a differencevisible image. That is, the difference actual object image generationunit 130 generates a difference actual object image which is an image ofthe difference between the actual object image at a certain timeacquired by the actual object image acquisition unit 110 and thebackground actual object image input from the background actual objectimage generation unit 120 (S107), and outputs the generated differenceactual object image at the certain time. In short, the difference actualobject image generation unit 130 generates a difference actual objectimage which is an image of the difference between the actual objectimage which is an image of an actual object obtained by photographing acertain range with the actual object camera 210 for photographing anactual object and a background actual object image which is an actualobject image of the background of the certain range. The generateddifference actual object image is input to the heat trace regionextraction unit 17. The difference actual object image generation unit130 may operate on the actual object image at the time desired to beprocessed by the heat trace region extraction unit 17, which will bedescribed later.

<Thermal Camera 22>

The thermal camera 22 is the same as the thermal camera 22 of the firstembodiment. That is, the thermal camera 22 is a camera for photographingthe heat emitted by the actual object. The thermal camera 22 photographsan image of the heat emitted by an actual object in the samephotographing range as the actual object camera 210, more specifically,a thermal image which is an image composed of an image of the heatradiated by the actual object existing in the photographing rangeacquired from the thermal camera 22 side. The thermal image photographedby the thermal camera 22 is input to the thermal image acquisition unit14.

<Thermal Image Acquisition Unit 14>

The thermal image acquisition unit 14 is the same as the thermal imageacquisition unit 14 of the first embodiment. That is, the thermal imageacquisition unit 14 acquires the thermal image photographed by thethermal camera 22 (S101) and outputs the acquired thermal image. Thethermal image acquired by the thermal image acquisition unit 14 is inputto the background thermal image generation unit 15 and/or the differencethermal image generation unit 16.

<Background Thermal Image Generation Unit 15>

The background thermal image generation unit 15 is the same as thebackground thermal image generation unit 15 of the first embodiment.That is, the background thermal image generation unit 15 generates abackground thermal image which is a thermal image of the background inthe photographing range based on the thermal image acquired by thethermal image acquisition unit 14 (S104), and outputs the generatedbackground thermal image. The thermal image of the background in thephotographing range is an image of the heat emitted by an objectexisting as the background in the photographing range, and morespecifically, an image composed of an image of the heat emitted by theobject existing as the background in the photographing range acquiredfrom the thermal camera 22 side. The generated background thermal imageis input to the difference thermal image generation unit 16.

<Difference Thermal Image Generation Unit 16>

The difference thermal image generation unit 16 is the same as thedifference thermal image generation unit 16 of the first embodiment.That is, the difference thermal image generation unit 16 generates adifference thermal image which is an image of the difference between thethermal image at a certain time acquired by the thermal imageacquisition unit 14 and the background thermal image input from thebackground thermal image generation unit 15 (S107), and outputs thegenerated difference thermal image at the certain time. In short, thedifference thermal image generation unit 16 generates a differencethermal image which is an image of the difference between a thermalimage which is an image of the heat emitted by an actual object obtainedby photographing the same certain range as the actual object camera 210by the thermal camera 22 for photographing the heat emitted by theactual object and a background thermal image which is a thermal image ofthe background in the certain range. The generated difference thermalimage is input to the heat trace region extraction unit 17. Thedifference thermal image generation unit 16 may operate on a thermalimage at the time desired to be processed by the heat trace regionextraction unit 17, which will be described later.

<Heat Trace Region Extraction Unit 17>

The heat trace region extraction unit 17 is the same as the heat traceregion extraction unit 17 of the first embodiment, except that it uses adifference actual object image instead of a difference visible image andthat it uses a background actual object image instead of a backgroundvisible image. That is, the heat trace region extraction unit 17extracts a heat trace region by removing the region of the actual objectfrom the thermal image based on the difference actual object image at acertain time generated by the difference actual object image generationunit 130 and the difference thermal image at the certain time generatedby the difference thermal image generation unit 16 (S108) and outputsthe information indicating the extracted heat trace region at thecertain time. Specifically, the heat trace region extraction unit 17extracts, as a heat trace region, a region which is not similar to anyof one or more difference actual object regions among one or moredifference thermal regions, using each region different from thebackground thermal image in the difference actual object image as adifference actual object region and using each region different from thebackground thermal image in the difference thermal image as a differencethermal region. The information indicating the extracted heat traceregion is input to the heat trace region output unit 18. The heat traceregion extraction unit 17 generates, for example, a binary image inwhich the heat trace region portion is white and the rest is black asinformation indicating the heat trace region, and outputs the generatedbinary image to the heat trace region output unit 18. The heat traceregion extraction unit 17 may also output the background actual objectimage to the heat trace region output unit 18. The heat trace regionextraction unit 17 may operate on the difference actual object image andthe difference thermal image at the time desired to be processed by theheat trace region extraction unit 17. The time desired to be processedby the heat trace region extraction unit 17 may be each time atpredetermined intervals, or may be a time designated by an operator orthe like of the heat trace region extraction device 10.

<Heat Trace Region Output Unit 18>

The heat trace region output unit 18 is the same as the heat traceregion output unit 18 of the first embodiment, except that it uses abackground actual object image instead of the background visible image.The heat trace region output unit 18 outputs information indicating theheat trace region so that the user can confirm it (S109). For example,the heat trace region output unit 18 outputs an image obtained bycombining white pixels of a binary image, which is an example ofinformation indicating a heat trace region, on a background actualobject image. For example, the heat trace region output unit 18 projectsa binary image showing the heat trace region to the photographing rangein the environment after performing alignment appropriately using aprojector or the like. The heat trace region output unit 18 may startoutputting information indicating the heat trace region when, forexample, information indicating the heat trace region is input. The heattrace region output unit 18 may, for example, continue to outputinformation indicating the heat trace region for a predetermined timeand end the output after the predetermined time has elapsed.Alternatively, the heat trace region output unit 18 may continue tooutput the information indicating the heat trace region until it isinstructed to end the output by the operator or the like of the heattrace region extraction device 10 and may end the output of theinformation indicating the heat trace region according to theinstruction.

Second Embodiment

The heat trace region extraction device and method of the firstembodiment and the modified example of the first embodiment can informthe user of a region where the temperature has risen due to contact witha person, for example, a region where a virus or the like may beattached. However, even in regions where the temperature has risen dueto contact with a person, regions that have been disinfected withalcohol or the like after being touched by people are regions wherethere is a low possibility of being infected with a virus or the likeeven if they are touched by people. The heat trace region extractiondevice and method of the second embodiment extract regions having a lowpossibility of being infected with a virus or the like even if they aretouched by such a person.

As shown in FIG. 9 , the heat trace region extraction system 1 of thesecond embodiment includes a heat trace region extraction device 10, anactual object camera 210, and a thermal camera 22. As shown in FIG. 9 ,the heat trace region extraction device 10 of the second embodimentincludes, for example, an actual object image acquisition unit 110, abackground actual object image generation unit 120, a difference actualobject image generation unit 130, a thermal image acquisition unit 14, abackground thermal image generation unit 15, a difference thermal imagegeneration unit 16, a difference cold image generation unit 161, a heattrace region extraction unit 17, a cold trace region extraction unit171, and an information output unit 181. The actual object camera 210and the thermal camera 22 are connected to the heat trace regionextraction device 10 of the second embodiment, and the images taken bythe actual object camera 210 and the thermal camera 22 are input to theheat trace region extraction device 10.

The heat trace region extraction method of the second embodiment isrealized by each unit of the heat trace region extraction deviceexecuting the processing of steps S101 to S1091 shown in FIG. 10 andbelow. Hereinafter, portions different from the modified example of thefirst embodiment will be mainly described. Duplicate explanations willbe omitted as appropriate for the same portions as those in the modifiedexample of the first embodiment.

<Actual Object Camera 210>

The actual object camera 210 is the same as the actual object camera 210of the modified example of the first embodiment. The actual objectcamera 210 photographs an image of an actual object in a certainphotographing range. The actual object image photographed by the actualobject camera 210 is input to the actual object image acquisition unit110.

<Actual Object Image Acquisition Unit 110>

The actual object image acquisition unit 110 is the same as the actualobject image acquisition unit 110 of the modified example of the firstembodiment. That is, the actual object image acquisition unit 110acquires the actual object image photographed by the actual objectcamera 210 (S101) and outputs the acquired actual object image. Theactual object image acquired by the actual object image acquisition unit110 is input to the background actual object image generation unit 120and/or the difference actual object image generation unit 130.

<Background Actual Object Image Generation Unit 120>

The background actual object image generation unit 120 is the same asthe background actual object image generation unit 120 of the modifiedexample of the first embodiment. That is, the background actual objectimage generation unit 120 generates a background actual object imagewhich is an actual object image of the background in the photographingrange based on the actual object image acquired by the actual objectimage acquisition unit 110 (S104) and outputs the generated backgroundactual object image. The generated background actual object image isinput to the difference actual object image generation unit 130.

<Difference Actual Object Image Generation Unit 130>

The difference actual object image generation unit 130 is the same asthe difference actual object image generation unit 130 of the modifiedexample of the first embodiment. That is, the difference actual objectimage generation unit 130 generates a difference actual object imagewhich is an image of the difference between the actual object image at acertain time acquired by the actual object image acquisition unit 110and the background actual object image input from the background actualobject image generation unit 120 (S107), and outputs the generateddifference actual object image at the certain time. In short, thedifference actual object image generation unit 130 generates adifference actual object image which is an image of the differencebetween the actual object image which is an image of an actual objectobtained by photographing a certain range with the actual object camerafor photographing an actual object and a background actual object imagewhich is an actual object image of the background of the certain range.In the second embodiment, the difference actual object image generatedby the difference actual object image generation unit 130 is input notonly to the heat trace region extraction unit 17 but also to the coldtrace region extraction unit 171. The difference actual object imagegeneration unit 130 may operate on an actual object image at the timedesired to be processed by the heat trace region extraction unit 17described later and an actual object image at the time desired to beprocessed by the cold trace region extraction unit 171 described later.

<Thermal Camera 22>

The thermal camera 22 is the same as the thermal camera 22 of themodified example of the first embodiment. That is, the thermal camera 22is a camera for photographing the heat emitted by the actual object. Thethermal camera 22 photographs an image of the heat emitted by the actualobject in the same photographing range as the actual object camera 210.The thermal image photographed by the thermal camera 22 is input to thethermal image acquisition unit 14.

<Thermal Image Acquisition Unit 14>

The thermal image acquisition unit 14 is the same as the thermal imageacquisition unit 14 of the modified example of the first embodiment.That is, the thermal image acquisition unit 14 acquires the thermalimage photographed by the thermal camera 22 (S101) and outputs theacquired thermal image. The thermal image acquired by the thermal imageacquisition unit 14 is input to the background thermal image generationunit 15 and/or the difference thermal image generation unit 16 and thedifference cold image generation unit 161.

<Background Thermal Image Generation Unit 15>

The background thermal image generation unit 15 is the same as thebackground thermal image generation unit 15 of the modified example ofthe first embodiment. That is, the background thermal image generationunit 15 generates a background thermal image which is a thermal image ofthe background in the photographing range based on the thermal imageacquired by the thermal image acquisition unit 14 (S104), and outputsthe generated background thermal image. The generated background thermalimage is input to the difference thermal image generation unit 16 andthe difference cold image generation unit 161.

<Difference Thermal Image Generation Unit 16>

The difference thermal image generation unit 16 is the same as thedifference thermal image generation unit 16 of the modified example ofthe first embodiment. The difference thermal image generation unit 16generates a difference thermal image by extracting a region (referred toas “high temperature region”) having a high temperature with respect tothe background thermal image input from the background thermal imagegeneration unit 15 from the thermal image at a certain time acquired bythe thermal image acquisition unit 14 (S107) and outputs the generateddifference thermal image at the certain time. That is, the differencethermal image generation unit 16 generates a difference thermal imagewhich is an image including the region of the difference between thethermal image and the background thermal image, the region where thetemperature in the thermal image is higher than the temperature in thebackground thermal image as a high temperature region. For example, thedifference thermal image generation unit 16 generates a differencethermal image in which, for each pixel, the pixel value of thedifference thermal image is set to 1 in a case where the differencebetween the pixel value of the thermal image and the pixel value of thebackground thermal image is equal to or more than a certain thresholdvalue and the temperature indicated by the pixel value of the thermalimage is higher than the temperature indicated by the pixel value of thebackground thermal image, and the pixel value of the difference thermalimage is set to 0 in other cases (that is, a case where the temperatureindicated by the pixel value of the thermal image is higher than thetemperature indicated by the pixel value of the background thermal imageand the difference between the pixel value of the thermal image and thepixel value of the background thermal image is less than the certainthreshold value and a case where the temperature indicated by the pixelvalue of the thermal image is not higher than the temperature indicatedby the pixel value of the background thermal image). The generateddifference thermal image is input to the heat trace region extractionunit 17. The difference thermal image generation unit 16 may operate ona thermal image at the time desired to be processed by the heat traceregion extraction unit 17 described later.

<Difference Cold Image Generation Unit 161>

The difference cold image generation unit 161 is the same as thedifference thermal image generation unit 16 of the modified example ofthe first embodiment, except that it generates a difference cold image,which is an image of the difference between the thermal image and thebackground thermal image and is an image of a region where thetemperature is lower than the background, instead of the differencethermal image. The difference cold image generation unit 161 generates adifference cold image by extracting a region (referred to as “lowtemperature region”) having a low temperature with respect to thebackground thermal image input from the background thermal imagegeneration unit 15 from the thermal image at a certain time acquired bythe thermal image acquisition unit 14 (S107) and outputs the generateddifference cold image at the certain time. That is, the difference coldimage generation unit 161 generates a difference cold image which is animage including the region of the difference between the thermal imageand the background thermal image, the region where the temperature inthe thermal image is lower than the temperature in the backgroundthermal image as a low temperature region. For example, the differencecold image generation unit 161 generates a difference cold image inwhich, for each pixel, the pixel value of the difference cold image isset to 1 in a case where the difference between the pixel value of thethermal image and the pixel value of the background thermal image isequal to or more than a certain threshold value and the temperatureindicated by the pixel value of the thermal image is lower than thetemperature indicated by the pixel value of the background thermalimage, and the pixel value of the difference thermal image is set to 0in other cases (that is, a case where the temperature indicated by thepixel value of the thermal image is lower than the temperature indicatedby the pixel value of the background thermal image and the differencebetween the pixel value of the thermal image and the pixel value of thebackground thermal image is less than the certain threshold value and acase where the temperature indicated by the pixel value of the thermalimage is not lower than the temperature indicated by the pixel value ofthe background thermal image). The generated difference cold image isinput to the cold trace region extraction unit 171. The difference coldimage generation unit 161 may operate on a thermal image at the timedesired to be processed by the cold trace region extraction unit 171described later.

<Heat Trace Region Extraction Unit 17>

The heat trace region extraction unit 17 is the same as the heat traceregion extraction unit 17 of the modified example of the firstembodiment. That is, the heat trace region extraction unit 17 extracts aheat trace region by removing the region of the actual object from thehigh temperature region included in the difference thermal image basedon the difference actual object image generated by the difference actualobject image generation unit 130 and the difference thermal imagegenerated by the difference thermal image generation unit 16 (S108) andoutputs the information indicating the extracted heat trace region.Specifically, the heat trace region extraction unit 17 extracts, as aheat trace region, a region which is not similar to any of one or moredifference actual object regions among one or more difference thermalregions, using each region different from the background thermal imagein the difference actual object image as a difference actual objectregion and using each region different from the background thermal imagein the difference thermal image as a difference thermal region. Theinformation indicating the extracted heat trace region is input to theinformation output unit 181. The heat trace region extracted by the heattrace region extraction unit 17 of the second embodiment is a region inwhich the temperature has risen due to contact with an actual object,and is, for example, a region touched by a person among objects existingas the background in the photographing range. The heat trace regionextraction unit 17 generates, for example, a binary image in which theheat trace region portion is white and the rest is black as informationindicating the heat trace region, and outputs the generated binary imageto the information output unit 181. The heat trace region extractionunit 17 may operate on the difference actual object image and thedifference thermal image at the time desired to be processed by the heattrace region extraction unit 17. The time desired to be processed by theheat trace region extraction unit 17 may be each time at predeterminedintervals, or may be a time designated by an operator or the like of theheat trace region extraction device 10.

<Cold Trace Region Extraction Unit 171>

The cold trace region extraction unit 171 is the same as the heat traceregion extraction unit 17 of the modified example of the firstembodiment except that it performs processing using a difference coldimage instead of a difference thermal image, and that it extracts a coldtrace region which is a region where the temperature of the heat emittedby the actual object is lowered instead of a heat trace region. That is,the cold trace region extraction unit 171 extracts a cold trace regionby removing the region of an actual object from a low temperature regionincluded in the difference cold image based on the difference actualobject image generated by the difference actual object image generationunit 130 and the difference cold image generated by the difference coldimage generation unit 161 (S1081) and outputs information indicating theextracted cold trace region. Specifically, the cold trace regionextraction unit 171 extracts, as a cold trace region, a region which isnot similar to any of one or more difference actual object regions amongone or more difference cold regions, using each region different fromthe background thermal image in the difference actual object image as adifference actual object region and using each region different from thebackground thermal image in the difference cold image as a differencecold region. The information indicating the extracted cold trace regionis input to the information output unit 181. The cold trace regionextraction unit 171 generates, for example, a binary image in which thecold trace region portion is white and the rest is black as informationindicating the cold trace region, and outputs the generated binary imageto the information output unit 181. The cold trace region extractionunit 171 may operate on the difference actual object image and thedifference cold image at the time desired to be processed by the coldtrace region extraction unit 171. The time desired to be processed bythe cold trace region extraction unit 171 may be each time atpredetermined intervals, or may be a time designated by an operator orthe like of the heat trace region extraction device 10.

The cold trace region extracted by the cold trace region extraction unit171 is a region where the temperature has decreased due to contact withthe actual object. When disinfection is performed by wiping withalcohol, the temperature of the wiped region is lowered by the heat ofvaporization of alcohol. Even when disinfection is performed by wipingwith water, chemicals, and the like instead of alcohol, the temperatureof the wiped region generally decreases due to the heat of vaporization.Therefore, the cold trace region extracted by the cold trace regionextraction unit 171 is a region where the temperature has decreased dueto disinfection with alcohol or the like, and is a region where there isa low possibility of being infected with a virus or the like even if aperson touches it. That is, by the processing of the cold trace regionextraction unit 171, it is possible to extract a region that is unlikelyto be infected with a virus or the like even if it is touched by aperson.

In addition to the disinfected region, the region such as a bottlecontaining the disinfectant used for disinfection may also appear in thedifference cold image obtained by the difference cold image generationunit 161 as the region where the temperature in the thermal image islower than the temperature in the background thermal image. For example,when the background actual object image is the image of FIG. 11(a), theactual object image is the image of FIG. 11(b), the difference actualobject image is the image of FIG. 11(c), and the background thermalimage is the image of FIG. 11(d) and the thermal image is the image ofFIG. 11(e), the difference cold image obtained by the difference coldimage generation unit 161 may be the image of FIG. 11(f). In thedifference cold image of FIG. 11(f), not only the disinfected region R1but also the bottle region R2 appears as a region where the temperaturehas decreased. Even in such a case, by the process of the cold traceregion extraction unit 171, that is, by the process of removing theregion of the actual object shown in the difference actual object imagefrom the low temperature region shown in the difference cold image, onlythe disinfected region R1 can be extracted as a cold trace region asshown in 11(g).

<Information Output Unit 181>

The information output unit 181 uses at least the information indicatingthe cold trace region to output at least the information indicating thecold trace region so that the user can confirm it (S1091). Theinformation output unit 181 operates as in the first and second examplesbelow, for example.

<<First Example of Information Output Unit 181>>

The information output unit 181 of the first example uses and outputsthe information indicating the heat trace region and the informationindicating the cold trace region so that the user can confirm at leastone of the information indicating the heat trace region and theinformation indicating the cold trace region. More specifically, theinformation output unit 181 of the first example continues to output theinformation indicating the heat trace region once extracted by the heattrace region extraction unit 17 so that the user can confirm it,determines whether the heat trace region overlaps with the cold traceregion at a time later than the time corresponding to the heat traceregion, and stops outputting all or part of the information indicatingthe heat trace region according to the determination result. That is,the information output unit 181 of the first example starts outputtingthe information indicating the heat trace region when the heat traceregion is extracted by the heat trace region extraction unit 17,determines whether the cold trace region overlaps with the heat traceregion when the cold trace region is extracted by the cold trace regionextraction unit 171, and ends the output of all or part of theinformation indicating the heat trace region when it is determined thatthey overlap. Hereinafter, an example of the processing of theinformation output unit 181 of the first example will be described.

First, when the heat trace region is extracted by the heat trace regionextraction unit 17, the information output unit 181 outputs informationindicating the heat trace region so that the user can confirm it in thesame manner as the heat trace region output unit 18 of the firstembodiment. For example, the information output unit 181 displays theinformation indicating the heat trace region so that the user canconfirm it by appropriately aligning a binary image, which isinformation indicating whether it is a heat trace region as shown inFIG. 12(a), and then projecting it onto a photographing range in theenvironment. In this example, the information indicating the heat traceregion is a binary image. In FIG. 12(a) and FIG. 12(b) used in thefollowing description, the region painted with the dot pattern is a heattrace region, and the region painted with white is not a heat traceregion.

After that, it is assumed that a cold trace region is extracted by thecold trace region extraction unit 171. An example of this cold traceregion is indicated by a broken line in FIG. 12(b). The broken line inFIG. 12(b) indicates the edge of the cold trace region. The regionsurrounded by the broken line in FIG. 12(b) is the cold trace region.When the cold trace region is extracted by the cold trace regionextraction unit 171, the information output unit 181 determines whetherthe heat trace region overlaps with the cold trace region for each ofthe heat trace regions extracted in the past.

For example, the information output unit 181 calculates the match rate,which is an index indicating how much of the heat trace region is thecold trace region. For example, the match rate=(the number of pixels inwhich the heat trace region and the cold trace region overlap)/(thenumber of pixels in the heat trace region). The information output unit181 stops the output of the information indicating the heat trace regionfor the heat trace region where the match rate is equal to or higherthan a predetermined threshold value. For example, when the heat traceregion and the cold trace region are in the state illustrated in FIG.12(b), the binary image illustrated in FIG. 12(c) is projected onto thephotographing range in the environment. That is, the display of the heattrace region is stopped. This predetermined threshold value is a realnumber close to 1, for example 0.9. As described above, the informationoutput unit 181 may stop the output of all the information indicatingthe heat trace region according to the determination result of whetherthe heat trace region overlaps with the cold trace region.

For example, the information output unit 181 stops the output ofinformation indicating the heat trace region for the portion of the heattrace region that overlaps with the cold trace region. For example, whenthe heat trace region and the cold trace region are in the stateillustrated in FIG. 12(b), the binary image illustrated in FIG. 12(d) isprojected onto the photographing range in the environment. In otherwords, the information output unit 181 may display only the regionexcluding the region where the heat trace region overlaps with the coldtrace region from the heat trace region. As described above, theinformation output unit 181 may stop the output of a part of theinformation indicating the heat trace region according to thedetermination result of whether the heat trace region overlaps with thecold trace region.

Even if a person infected with a virus or the like touches a region inthe past and the virus or the like has been attached to the region, theregion disinfected with alcohol is a region where there is a lowpossibility of being infected with the virus or the like even if it istouched by people. The heat trace region is a region where thetemperature has risen due to contact with a person, and is considered tobe a region where a virus or the like may be attached. On the otherhand, the cold trace region is a region where the temperature hasdecreased due to disinfection with alcohol or the like, and isconsidered to be a region where there is low possibility of beinginfected with the virus or the like even if it is touched by people. Asin the first example of the information output unit 181, the informationindicating the heat trace region once extracted is continuously outputso that the user can confirm it, but the output of a part of theinformation indicating the heat trace region is stopped according to thedetermination result of whether the heat trace region overlaps with acold trace region extracted later. By doing so, the user can bepresented with only the region that may be infected with a virus or thelike.

<<Second Example of Information Output Unit 181>>

The information output unit 181 of the second example uses and outputsthe information indicating the cold trace region so that the user canconfirm the information indicating the cold trace region. Morespecifically, the information output unit 181 of the second exampleoutputs only the information indicating the cold trace region extractedby the cold trace region extraction unit 171 so that the user canconfirm it. For example, the information output unit 181 outputs animage obtained by combining white pixels of a binary image, which is anexample of information indicating a cold trace region, on a backgroundactual object image. For example, the information output unit 181projects a binary image showing a cold trace region onto a photographingrange in the environment after performing alignment appropriately usinga projector or the like. As a result, the user can know the region wherethere is a low possibility of being infected with a virus or the likeeven if the user touches the region disinfected with alcohol or thelike. The information output unit 181 may start outputting theinformation indicating the cold trace region, for example, when theinformation indicating the cold trace region is input.

When the information output unit 181 performs the operation of thesecond example, the heat trace region extraction device 10 does not needto obtain information indicating the heat trace region. In such a case,the heat trace region extraction device 10 may not include thedifference thermal image generation unit 16 and the heat trace regionextraction unit 17. The heat trace region extraction device 10 that doesnot include the difference thermal image generation unit 16 and the heattrace region extraction unit 17 can be said to be a cold trace regionextraction device.

Third Embodiment

The heat trace region extraction device and method of the firstembodiment can inform the user of a region where the temperature hasrisen due to contact with a person and a region where a virus or thelike may be attached. However, even a region where the temperature hasrisen due to contact with a person with a low body temperature is aregion where it is unlikely that the virus or the like is attached sinceit is unlikely that the touched person is infected with a virus or thelike. On the other hand, a region where the temperature has risen due tocontact with a person with a high body temperature is a region wherethere is a high possibility that the virus or the like is attachedbecause the touched person is likely to be infected with the virus orthe like. From these facts, in order to more appropriately inform theuser of the region where a virus or the like may be attached, it may bebetter if it is possible to identify a person or the body temperaturethat causes the heat trace corresponding to the heat trace region aswell as extracting the heat trace region which is the region where thetemperature has risen due to contact with a person. The heat traceregion extraction device and method of the third embodiment identify thetemperature that causes the heat trace corresponding to the heat traceregion.

As shown in FIG. 13 , the heat trace region extraction system 1 of thethird embodiment includes a heat trace region extraction device 10, anactual object camera 210, and a thermal camera 22. As shown in FIG. 13 ,the heat trace region extraction device 10 of the third embodimentincludes, for example, an actual object image acquisition unit 110, abackground actual object image generation unit 120, a difference actualobject image generation unit 130, a thermal image acquisition unit 14, abackground thermal image generation unit 15, a difference thermal imagegeneration unit 16, a heat trace region extraction unit 17, a heat tracecause identification unit 31, and an information output unit 181. Theactual object camera 210 and the thermal camera 22 are connected to theheat trace region extraction device 10 of the third embodiment, and theimages taken by the actual object camera 210 and the thermal camera 22are input to the heat trace region extraction device 10.

The heat trace region extraction method of the third embodiment isrealized by each unit of the heat trace region extraction deviceexecuting the processing of steps S101 to S109 shown in FIG. 14 andbelow. Hereinafter, portions different from the modified example of thefirst embodiment will be mainly described. Duplicate explanations willbe omitted as appropriate for the same portions as those in the modifiedexample of the first embodiment.

The actual object camera 210, the actual object image acquisition unit110, the background actual object image generation unit 120, thedifference actual object image generation unit 130, the thermal camera22, the thermal image acquisition unit 14, the background thermal imagegeneration unit 15, and the difference thermal image generation unit 16are the same as the actual object camera 210, the actual object imageacquisition unit 110, the background actual object image generation unit120, the difference actual object image generation unit 130, the thermalcamera 22, the thermal image acquisition unit 14, the background thermalimage generation unit 15, and the difference thermal image generationunit 16 of the modified example of the first embodiment, respectively.However, the difference thermal image generated by the differencethermal image generation unit 16 and the thermal image acquired by thethermal image acquisition unit 14 are further input to the heat tracecause identification unit 31.

In addition to the operation of the heat trace region extraction unit 17of the modified example of the first embodiment, the heat trace regionextraction unit 17 generates and outputs a difference thermal regionimage showing one difference thermal region as a difference and a heattrace region image showing only one heat trace region as a differenceand outputs time information corresponding to each heat trace regionimage. The heat trace region extraction unit 17 assigns a file name toeach image and also outputs the file name. The difference thermal regionimage, the heat trace region image, the time information correspondingto the heat trace region image, and the file name of each image obtainedby the heat trace region extraction unit 17 are input to the heat tracecause identification unit 31.

When one difference thermal region is obtained, the heat trace regionextraction unit 17 generates an image showing the one difference thermalregion as a difference as a difference thermal region image. Further,when p is a positive integer of 2 or more and p difference thermalregions separated from each other are obtained, the heat trace regionextraction unit 17 generates p images each showing only one differencethermal region different among the p difference thermal regions as adifference as difference thermal region images. The heat trace regionextraction unit 17 generates, for example, a binary image in which thepixel value of each pixel in the difference thermal region is 1 and thepixel value of each pixel in a region other than the difference thermalregion is 0 as a difference thermal region image.

When one heat trace region is obtained, the heat trace region extractionunit 17 generates an image showing the one heat trace region as adifference as a heat trace region image. Further, when q is a positiveinteger of 2 or more and q heat trace regions separated from each otherare obtained, the heat trace region extraction unit 17 generates qimages each showing only one heat trace region different among the qheat trace regions as a region as heat trace region images. The heattrace region extraction unit 17 generates, for example, a binary imagein which the pixel value of each pixel in the heat trace region is 1 andthe pixel value of each pixel in a region other than the heat traceregion is 0 as a heat trace region image.

In the third embodiment, a heat trace table is created based on the heattrace region image and the time information corresponding to the heattrace region image obtained by the heat trace region extraction unit 17.The heat trace table is stored in the auxiliary storage device 102. Anexample of a heat trace table is shown in FIG. 15 . In the heat tracetable of FIG. 15 , records for each heat trace region at each timeextracted by the heat trace region extraction unit 17 are recorded, andthe records for each heat trace region include the time corresponding tothe heat trace region image, the file name of the heat trace regionimage, and the group ID to which the heat trace region belongs. Thegroup ID to which each heat trace region belongs in the heat trace tableis generated by the heat trace cause identification unit 31 describedlater.

The heat trace region at the time “2021-08-10 10:50:48.000” in FIG. 15is the heat trace region extracted by the process corresponding to thelatest time “2021-08-10 10:50:48.000” in the heat trace regionextraction unit 17 and is not yet processed by the heat trace causeidentification unit 31. Therefore, the group ID to which the heat traceregion belongs is not assigned to the record of the heat trace region atthe time “2021-08-10 10:50:48.000” in FIG. 15 . The heat trace causeidentification unit 31 performs a process of identifying the cause ofthe corresponding heat trace on the heat trace region extracted by theprocess corresponding to the latest time in the heat trace regionextraction unit 17 and assigns a group ID during the process.

<Heat Trace Cause Identification Unit 31>

The heat trace cause identification unit 31 performs a firstidentification process of identifying a heat trace region due to a causedifferent from a cause by which the heat trace region extracted by theprocess corresponding to the past time became a heat trace among theheat trace regions extracted by the process corresponding to the latesttime, a second identification process of identifying a causal regionwhich is a region which became the cause by which the heat trace regionidentified in the first identification process became the heat traceusing the difference thermal image extracted by the processcorresponding to the past time, of the heat trace region extraction unit17, and a third identification process of identifying a temperature ofthe causal region identified in the second identification process from athermal image corresponding to the difference thermal image used foridentifying the causal region (S310). The temperature of the causalregion identified in the third identification process (that is, thetemperature of the causal region of the heat trace corresponding to eachheat trace region) is input to the information output unit 181.

Specifically, the second identification process performed by the heattrace cause identification unit 31 is a process of identifying adifference thermal image of a time (past time) earlier than thedifference thermal image corresponding to the heat trace regionidentified in the first identification process, the difference thermalimage satisfying a condition that a region corresponding to the heattrace region identified in the first identification process is included,as a difference thermal region, in all difference thermal images rangingfrom the difference thermal images corresponding to the heat traceregion identified in the first identification process to the differencethermal images of the earlier time (past time), and the differencethermal image including a difference thermal region in which a regiondifferent from the region corresponding to the heat trace regionidentified in the first identification process is connected to a regioncorresponding to the heat trace region identified in the firstidentification process, and identifying the different region in theidentified difference thermal image as a causal region which became thecause of the heat trace corresponding to the heat trace regionidentified in the first identification process.

Hereinafter, an example of the processing of the heat trace causeidentification unit 31 will be described. In the following description,the heat trace region extracted by the heat trace region extraction unit17 in the process corresponding to the latest time is referred to as anew heat trace region. In the following description, the heat traceregion extracted by the process corresponding to the time earlier thanthe latest time is also referred to as an old heat trace region.

The heat trace cause identification unit 31 performs the followingprocessing each time the heat trace region is extracted by theprocessing corresponding to the latest time in the heat trace regionextraction unit 17. That is, the heat trace cause identification unit 31performs the following processing on each new heat trace region.

First, the heat trace cause identification unit 31 determines a group towhich the new heat trace region belongs by performing the processing ofsteps S3101 to S3106 shown in FIG. 16 and below.

The heat trace cause identification unit 31 selects one old heat traceregion latest among the old heat trace regions that have not yet beenselected (S3101), and the processing proceeds to step S3102. The heattrace cause identification unit 31 selects, for example, one old heattrace region in which the group ID recorded in the heat trace table isdifferent from the group IDs of the old heat trace regions selected sofar and the time recorded in the heat trace table is the latest.Hereinafter, the old heat trace region selected by the heat trace causeidentification unit 31 is abbreviated as “selected old heat traceregion”.

The heat trace cause identification unit 31 determines whether thedifference between the time corresponding to the new heat trace regionand the time corresponding to the selected old heat trace region isequal to or less than a predetermined time (S3102). For example, theheat trace cause identification unit 31 may determine whether thedifference between the time included in the record of the new heat traceregion recorded in the heat trace table and the time included in therecord of the selected old heat trace region recorded in the heat tracetable is equal to or less than a predetermined time. The predeterminedtime is longer than an average time in which the temperature of the heattrace region decreases and becomes the same as the temperature of thebackground. For example, the predetermined time is 10 seconds.

When the heat trace cause identification unit 31 determines in stepS3102 that the difference between the time corresponding to the new heattrace region and the time corresponding to the selected old heat traceregion is equal to or less than the predetermined time, the processingproceeds to step S3103. When it is determined in step S3102 that thedifference between the time corresponding to the new heat trace regionand the time corresponding to the selected old heat trace region is notless than or equal to the predetermined time, the processing proceeds tostep S3106. The processing of step S3106, which is a process performedwhen it is determined in step S3102 that the difference between the timecorresponding to the new heat trace region and the time corresponding tothe selected old heat trace region is not equal to or less than thepredetermined time, will be described later.

When the difference between the corresponding time of the new heat traceregion and the corresponding time of the selected old heat trace regionis less than or equal to the predetermined time, the heat trace causeidentification unit 31 determines whether the size of the region notincluded in the selected old heat trace region in the new heat traceregion is equal to or smaller than a predetermined size (S3103). Theheat trace cause identification unit 31 may make the determination ofstep S3103 using, for example, a heat trace region image of a fileidentified by the file name of a new heat trace region recorded in theheat trace table and a heat trace region image of a file identified bythe file name of a selected old heat trace region recorded in the heattrace table.

When the heat trace cause identification unit 31 determines in stepS3103 that the size of the region not included in the selected old heattrace region in the new heat trace region is not equal to or smallerthan the predetermined size, the processing proceeds to S3101. When theheat trace cause identification unit 31 determines in step S3103 thatthe size of the region not included in the selected old heat traceregion in the new heat trace region is equal to or less than thepredetermined size, the processing proceeds to step S3104.

When the size of the region not included in the selected old heat traceregion in the new heat trace region is less than or equal to thepredetermined size, the heat trace cause identification unit 31determines whether the temperature corresponding to the new heat traceregion is lower than the temperature corresponding to the selected oldheat trace region (S3104). The temperature corresponding to the new heattrace region can be obtained from the thermal image corresponding to thenew heat trace region. For example, the heat trace cause identificationunit 31 calculates the average value of the temperature indicated byeach pixel included in the region corresponding to the new heat traceregion in the thermal image corresponding to the heat trace region imageof the new heat trace region and uses the average value as thetemperature corresponding to the new heat trace region. Similarly, thetemperature corresponding to the selected old heat trace region can beobtained from the thermal image corresponding to the selected old heattrace region. For example, the heat trace cause identification unit 31calculates the average value of the temperature indicated by each pixelincluded in the region corresponding to the selected old heat traceregion in the thermal image corresponding to the heat trace region imageof the selected old heat trace region and uses the average value as thetemperature corresponding to the selected old heat trace region.

When the heat trace cause identification unit 31 determines in stepS3104 that the temperature corresponding to the new heat trace region isnot lower than the temperature corresponding to the selected old heattrace region, the processing proceeds to step S3101. When the heat tracecause identification unit 31 determines in step S3104 that thetemperature corresponding to the new heat trace region is lower than thetemperature corresponding to the selected old heat trace region, theprocessing proceeds to step S3105.

When the temperature corresponding to the new heat trace region is lowerthan the temperature corresponding to the selected old heat traceregion, the heat trace cause identification unit 31 determines that thenew heat trace region belongs to the same group as the selected old heattrace region (S3105). In this case, the heat trace cause identificationunit 31 records, for example, the same number as the group ID of therecord of the selected old heat trace region recorded in the heat tracetable as the group ID of the record of the new heat trace region in theheat trace table.

When the size of the region not included in the selected old heat traceregion in the new heat trace region is less than or equal to thepredetermined size, and the temperature corresponding to the new heattrace region is lower than the temperature corresponding to the selectedold heat trace region, the new heat trace region is considered to be theregion of the selected old heat trace region, which is reduced with thepassage of time and in which the temperature has decreased. Therefore,in this case, the heat trace cause identification unit 31 may determinethat the cause of the heat trace corresponding to the new heat traceregion is the same as the cause of the heat trace corresponding to theselected old heat trace region. This determination is performed in theprocesses from step S3103 to step S3105.

When it is determined in the processing of step S3102 that thedifference between the time corresponding to the new heat trace regionand the time corresponding to the selected old heat trace region is notless than or equal to the predetermined time, the heat trace causeidentification unit 31 determines that the new heat trace region belongsto a new group (S3106). That is, when it is determined in the processingof step S3102 that the difference between the time corresponding to thenew heat trace region and the time corresponding to the selected oldheat trace region is not equal to or less than the predetermined time,the heat trace cause identification unit 31 determines that the new heattrace region is a heat trace region due to a new cause different fromthe cause by which the heat trace region extracted by the processcorresponding to the past time became the heat trace. In this case, theheat trace cause identification unit 31 records, for example, a numberdifferent from the group ID of the record of any one of the old heattrace regions recorded in the heat trace table as the group ID of therecord of the new heat trace region in the heat trace table.

In general, heat traces often disappear within seconds of the generationof heat traces. Therefore, if the difference between the timecorresponding to the new heat trace region and the time corresponding tothe latest old heat trace region is not equal to or less than thepredetermined time, the new heat trace region is considered to begenerated based on a cause different from the cause of the heat tracecorresponding to the old heat trace region. The determination based onthis idea is the processing of step S3102 and step S3106.

As indicated by the broken line in FIG. 16 , the heat trace causeidentification unit 31 determines that the new heat trace region belongsto a new group even if there is no old heat trace region that has notyet been selected in the processing of step S3101 performed first, inother words, even if there is no old heat trace region to be selected(S3106). In this case, the heat trace cause identification unit 31records, for example, an arbitrary number in the heat trace table as thegroup ID of the record of the new heat trace region.

For example, when it is determined that the heat trace region at thetime “2021-08-10 10:50:48.000” in the heat trace table of FIG. 15belongs to a new group, “28” which is a new group ID is assigned to theheat trace region.

When it is determined that the new heat trace region belongs to a newgroup, the heat trace cause identification unit 31 performs theprocessing of steps S3107 to S3110 shown in FIG. 17 and below toidentify the causal region which is a region that caused the heat traceof the new heat trace region and estimate the temperature of the causalregion of the heat trace.

The heat trace cause identification unit 31 acquires the latestdifference thermal image of the difference thermal images that have notbeen selected yet among the difference thermal images of the timeearlier than the time corresponding to the new heat trace regiondetermined to belong to a new group (S3107) and the processing proceedsto step S3108. For example, in the first processing of step S3107, theheat trace cause identification unit 31 acquires a difference thermalimage one hour before the time corresponding to the new heat traceregion. For example, with k being a positive integer of 2 or more, inthe (k−1)-th and subsequent processing of step S3107, the heat tracecause identification unit 31 acquires a difference thermal image onehour before the time of the difference thermal image acquired in thek-th processing of step S3107.

In the processing of steps S3107 to S3110, the heat trace causeidentification unit 31 may use the difference thermal region image inwhich the difference thermal region is included in the regioncorresponding to the new heat trace region instead of the differencethermal image. When the heat trace cause identification unit 31 uses thedifference thermal region image instead of the difference thermal image,the difference thermal image generated by the difference thermal imagegeneration unit 16 may not be input to the heat trace causeidentification unit 31.

The heat trace cause identification unit 31 determines whether theacquired difference thermal image includes a difference thermal regionin which a region different from the region corresponding to the newheat trace region is connected to the region corresponding to the newheat trace region (S3108).

When the heat trace cause identification unit 31 determines that theacquired difference thermal image does not include a difference thermalregion in which a region different from the region corresponding to thenew heat trace region is connected to the region corresponding to thenew heat trace region, the processing proceeds to step S3107. When theheat trace cause identification unit 31 determines that the acquireddifference thermal image includes a difference thermal region in which aregion different from the region corresponding to the new heat traceregion is connected to the region corresponding to the new heat traceregion, the processing proceeds to step S3109.

When the difference thermal image includes a difference thermal regionin which a region different from the region corresponding to the newheat trace region (hereinafter, referred to as “connecting region” forconvenience) is connected to the region corresponding to the new heattrace region, the heat trace cause identification unit 31 identifies theconnecting region as the causal region that caused the heat trace of thenew heat trace region (S3109), and the processing proceeds to stepS3110.

The heat trace cause identification unit 31 estimates the temperature ofthe causal region from the temperature of the region corresponding tothe causal region of the thermal image corresponding to the differencethermal image (S3110). For example, the heat trace cause identificationunit 31 calculates the average value of the temperature indicated byeach pixel included in the region corresponding to the causal region inthe thermal image corresponding to the difference thermal image, anduses the average value as the temperature of the causal region. The heattrace cause identification unit 31 may use the median or maximum valueof the temperature indicated by each pixel included in the regioncorresponding to the causal region in the thermal image corresponding tothe difference thermal image as the temperature of the causal region.

In this way, the heat trace cause identification unit 31 performs thefollowing processing using the extracted heat trace region as the newheat trace region, and using the heat trace region extracted at a timeearlier than the time when the new heat trace region is extracted as theold heat trace region. (1) When the difference between the timecorresponding to the new heat trace region and the time corresponding tothe old heat trace region is less than or equal to a predetermined time,the size of a region not included in the old heat trace region in thenew heat trace region is equal to or less than a predetermined size, andthe temperature corresponding to the new heat trace region is lower thanthe temperature corresponding to the old heat trace region, it isdetermined that the new heat trace region belongs to the same group asthe old heat trace region. That is, it is determined that the cause bywhich the new heat trace region became the heat trace is the same as thecause by which the old heat trace region became the heat trace. (2) Inother case, it is determined that the new heat trace region belongs to anew group. That is, the cause by which the new heat trace region becamethe heat trace is different from the cause by which the old heat traceregion became the heat trace. The heat trace cause identification unit31 identifies a difference thermal image of a time earlier than thedifference thermal image corresponding to the new heat trace regiondetermined to belong to the new group, the difference thermal imagesatisfying a condition that a region corresponding to the new heat traceregion determined to belong to the new group is included as a differencethermal region, in all difference thermal images ranging from thedifference thermal image corresponding to the new heat trace regiondetermined to belong to the new group to the difference thermal image ofthe earlier time, the difference thermal image including a differencethermal region in which a region different from the region correspondingto the new heat trace region is connected to the region corresponding tothe new heat trace region, and identifies the different region in theidentified difference thermal image as a causal region which is a regionthat caused the heat trace corresponding to the new heat trace region.The heat trace cause identification unit estimates the temperature ofthe region identified as the causal region from the thermal imagecorresponding to the difference thermal image used for identifying thecausal region.

Hereinafter, an example of the processing of the heat trace causeidentification unit 31 will be described with reference to FIG. 18 .FIG. 18(a) is a schematic diagram of a background actual object imageand an actual object image at each time, FIG. 18(b) is a schematicdiagram of a background thermal image and a thermal image at each time,FIG. 18(c) is a schematic diagram of a difference actual object image ateach time, FIG. 18(d) is a schematic diagram of a difference thermalimage at each time, and FIG. 18(e) is a schematic diagram of an imageshowing a heat trace region at each time.

At time t0, it is assumed that there is a region where the person Atouches the wall and the temperature has risen. Further, it is assumedthat the person A moves away from the wall at time t1 and the person Agoes out of the photographing range at time t2. After that, it isassumed that the person B crosses the photographing range at time t3 andthe person B goes out of the photographing range at time t4. Assumingthat using τ=1, 2, 4, the heat trace region extracted at time t_(τ) willbe referred to as the heat trace region r_(τ). At time t3, the heattrace region is not extracted because the person B overlaps with theregion where the heat trace regions r1 and r2 were.

First, an example of identifying the region that caused the heat traceof the heat trace region r1 at time t1 will be described. In thisexample, the heat trace region r1 is referred to as a new heat traceregion r1. In this example, at time t1, since the heat trace region isnot extracted at the time earlier than the time t1, the information ofthe old heat trace region does not exist. Therefore, since the heattrace cause identification unit 31 cannot select the old heat traceregion (S3101), it is determined that the new heat trace region r1belongs to a new group (S3106). The heat trace cause identification unit31 acquires a difference thermal image at time t0, which is the latestdifference thermal image before time t1 (S3107), and determines whetherthe difference thermal image at time t0 includes a difference thermalregion in which a region different from the region corresponding to thenew heat trace region r1 is connected to the region corresponding to thenew heat trace region r1 (S3108). In this example, in the differencethermal image at time t0 (the image at time t0 in FIG. 18(d)), theregion different from the region corresponding to the new heat traceregion r1 (the region of person A) is included as a difference thermalregion connected to the region corresponding to the new heat traceregion r1. Therefore, the heat trace cause identification unit 31identifies the region of the person A in the difference thermal image attime t0 as the region that caused the heat trace in the new heat traceregion r1 (S3109).

Next, an example of identifying the region that caused the heat trace ofthe heat trace region r2 at time t2 will be described. In this example,the heat trace region r2 is referred to as a new heat trace region r2.

First, the heat trace cause identification unit 31 selects the heattrace region r1 extracted at time t1 as an old heat trace region(S3101). Hereinafter, the heat trace region r1 will be referred to as anold heat trace region r1. The heat trace cause identification unit 31determines whether the difference between the time t2 corresponding tothe new heat trace region r2 and the time t1 corresponding to the oldheat trace region r1 is equal to or less than a predetermined time(S3102). The heat trace cause identification unit 31 also determineswhether the size of the region not included in the old heat trace regionr1 in the new heat trace region r2 is equal to or smaller than apredetermined size (S3103). The heat trace cause identification unit 31also determines whether the temperature corresponding to the new heattrace region r2 is lower than the temperature corresponding to the oldheat trace region r1 (S3104).

In this example, it is assumed that the difference between the time t2corresponding to the new heat trace region r2 and the time t1corresponding to the old heat trace region r1 is less than or equal tothe predetermined time, the size of the region not included in the oldheat trace region r1 in the new heat trace region r2 is equal to orsmaller than the predetermined size, and the temperature correspondingto the new heat trace region r2 is lower than the temperaturecorresponding to the old heat trace region r1. Therefore, the heat tracecause identification unit 31 determines that the new heat trace regionr2 belongs to the same group as the old heat trace region r1 (S3105).

The fact that the new heat trace region r2 belongs to the same group asthe old heat trace region r1 means that the cause of the heat trace inthe new heat trace region r2 is the same as the cause of the heat tracein the old heat trace region r1. The region that caused the heat traceof the old heat trace region r1 has already been identified as theregion of person A in the above-mentioned example at time t1. Therefore,the region that caused the heat trace of the new heat trace region r2 isthe region of person A.

Next, an example of identifying the region that caused the heat trace ofthe heat trace region r4 at time t4 will be described. In this example,the heat trace region r4 is referred to as a new heat trace region r4.In this example, there are two persons who are candidates for the causeof the heat trace corresponding to the new heat trace region r4. Thesetwo persons are person A and person B. Therefore, whether the cause ofthe heat trace corresponding to the new heat trace region r4 is theperson A or the person B is a problem.

First, the heat trace cause identification unit 31 selects the heattrace region r2 extracted at time t2 as the old heat trace region(S3101). Hereinafter, the heat trace region r2 will be referred to as anold heat trace region r2. The heat trace cause identification unit 31determines whether the difference between the time t4 corresponding tothe new heat trace region r4 and the time t2 corresponding to the oldheat trace region r2 is less than or equal to a predetermined time(S3102). The heat trace cause identification unit 31 also determineswhether the size of the region not included in the old heat trace regionr2 in the new heat trace region r4 is equal to or smaller than thepredetermined size (S3103). The heat trace cause identification unit 31also determines whether the temperature corresponding to the new heattrace region r4 is lower than the temperature corresponding to the oldheat trace region r2 (S3104).

In this example, it is assumed that the difference between the time t4corresponding to the new heat trace region r4 and the time t2corresponding to the old heat trace region r2 is less than or equal tothe predetermined time, the size of the region not included in the oldheat trace region r2 in the new heat trace region r4 is equal to orsmaller than the predetermined size, and the temperature correspondingto the new heat trace region r4 is lower than the temperaturecorresponding to the old heat trace region r2. Therefore, the heat tracecause identification unit 31 determines that the new heat trace regionr4 belongs to the same group as the old heat trace region r2 (S3105).

The fact that the new heat trace region r4 belongs to the same group asthe old heat trace region r2 means that the new heat trace region r4belongs to the same group as the old heat trace region r1 assuming thatit is already identified in the above-mentioned example at time t2 thatthe old heat trace region r2 belongs to the same group as the old heattrace region r1. That is, the cause of the heat trace in the new heattrace region r4 is the same as the cause of the heat trace in the oldheat trace region r1. The region that caused the heat trace of the oldheat trace region r1 has already been identified as the region of personA in the above-mentioned example at time t1. Therefore, the region thatcaused the heat trace of the new heat trace region r4 is the region ofperson A.

As in this example, even if there are two or more persons who arecandidates for the cause of the heat trace in the heat trace region, theperson who causes the heat trace in the heat trace region can beidentified more appropriately by the processing of the heat trace causeidentification unit 31 described above.

<Information Output Unit 181>

The information output unit 181 receives the information indicating theheat trace region extracted by the heat trace region extraction unit 17,and the temperature of the causal region of the heat trace correspondingto each heat trace region identified by the heat trace causeidentification unit 31.

The information output unit 181 outputs the information indicating theheat trace region input from the heat trace region extraction unit 17 sothat the user can confirm it in an expression method determinedaccording to the temperature of the causal region input from the heattrace cause identification unit 31 (S109). For example, the informationoutput unit 181 displays the information indicating the heat traceregion extracted by the heat trace region extraction unit 17 in anexpression method determined according to the temperature identified bythe heat trace cause identification unit 31. For example, theinformation output unit 181 projects an image or video obtained byprocessing a binary image, in which the heat trace region portion, whichis information indicating the heat trace region, is white and the restis black, in accordance with the temperature of the causal region inputfrom the heat trace cause identification unit 31, to a photographingrange.

An example of the expression method according to the temperature of thecausal region is an expression method that is stronger as thetemperature increases. The expression method that is stronger as thetemperature increases is, in other words, an expression method thatattracts the user's attention as the temperature increases. An exampleof the expression method that is stronger as the temperature increasesis an expression method in which, when the temperature of the heat traceregion is higher than a predetermined threshold value, display ofinformation indicating that it is the heat trace region portion or theheat trace region blinks, and the speed of blinking the display of theinformation indicating that it is the heat trace region portion or theheat trace region increases as the temperature increases. For example,this example can be realized when the information output unit 181projects a video onto the photographing range such that the brightnessof the video is set to 0 in a region other than the heat trace regionand is set to a predetermined brightness in the heat trace region wherethe temperature is equal to or lower than a predetermined thresholdvalue, and the brightness changes periodically in the time directionlike a rectangular wave between the predetermined brightness and 0 at ahigher frequency as the temperature increases in the heat trace regionwhere the temperature is higher than the predetermined threshold value.

Another example of the expression method that is stronger as thetemperature increases is an expression method in which when thetemperature of the heat trace region is higher than a predeterminedthreshold value, the color of display of information indicating that itis the heat trace region portion or the heat trace region changes, andthe speed of changing the color of the display of information indicatingthat it is the heat trace region portion or the heat trace regionincreases as the temperature of the heat trace region increases. Anotherexample of the expression method that is stronger as the temperatureincreases is an expression method in which when the temperature of theheat trace region is higher than a predetermined threshold value, thedisplay of information indicating that it is the heat trace regionportion or the heat trace region vibrates, and the frequency ofvibrating of the display of information indicating that it is the heattrace region portion or the heat trace region increases as thetemperature of the heat trace region increases. Another example of theexpression method that is stronger as the temperature increases is anexpression method in which the display of information indicating that itis the heat trace region portion or the heat trace region is brighter asthe temperature of the heat trace region increases. Another example ofthe expression method that is stronger as the temperature increases isan expression method in which the duration of the portion of the displayindicating that it is the heat trace region portion or the heat traceregion increases as the temperature of the heat trace region increases.

Another example of the expression method that differs depending on thetemperature of the causal region is an expression method of displayingthe numerical value of the temperature together with the informationindicating that it is the heat trace region portion or the heat traceregion. Another example of the expression method that differs dependingon the temperature of the causal region is an expression method in whichthe information indicating that it is the heat trace region portion orthe heat trace region is indicated by a different color depending on thetemperature. In an example of this expression method, when thetemperature of the heat trace region is higher than a predeterminedthreshold value, information indicating that it is the heat trace regionportion or the heat trace region is displayed in a first color, and inother cases, the information indicating that it is the heat trace regionportion or the heat trace region is displayed in a second color. Forexample, the first color is red and the second color is blue.

The information output unit 181 may create a group information tableillustrated in FIG. 19 and display information with reference to thegroup information table and the heat trace table stored in the auxiliarystorage device 102. The group information table of FIG. 19 storesrecords corresponding to each heat trace region, and each recordincludes a group ID to which each heat trace region belongs, thetemperature of a region that caused the heat trace of each heat traceregion, and an ID for identifying a display method corresponding to eachheat trace region. In this example, the ID that identifies the displaymethod is represented by a number that increases as the temperatureincreases. When the ID is displayed in a stronger expression as thetemperature increases, the information output unit 181 may use anexpression method that becomes stronger as the ID for identifying thedisplay method increases.

Fourth Embodiment

The heat trace region extraction device and method of the firstembodiment can inform the user of a region where the temperature hasrisen due to contact with a person, in other words, a region where avirus such as a new coronavirus may be attached by projecting the regionusing a projector or the like. However, when the projector projects withvisible light, the visible light camera 21 also photographs the visiblelight projected by the projector, and a visible image is obtained inwhich light and shade due to the light projected by the projector issuperimposed on a person, an object, or the like existing in thephotographing range. In particular, when the output of the projector islarge, the light and shade due to the light projected by the projectoris strongly superimposed on the image of a person or an object existingin the photographing range. Therefore, there is a possibility that theheat trace region cannot be appropriately extracted by the heat traceregion extraction device and method of the first embodiment.

In the heat trace region extraction device and method of the fourthembodiment, an actual object camera 210 that does not photograph thewavelength band of visible light projected by the projector is used as acamera for photographing an actual object instead of the visible lightcamera 21. Therefore, the wavelength band of the electromagnetic wavesobtained by the actual object camera 210 and the wavelength band of thevisible light projected by the projector do not overlap, and the lightprojected by the projector does not affect the extraction of the heattrace region.

As shown in FIG. 20 , the heat trace region extraction system 1 of thefourth embodiment includes a heat trace region extraction device 10, anactual object camera 210, and a thermal camera 22. As shown in FIG. 20 ,the heat trace region extraction device 10 of the fourth embodimentincludes, for example, an actual object image acquisition unit 110, abackground actual object image generation unit 120, a difference actualobject image generation unit 130, a thermal image acquisition unit 14, abackground thermal image generation unit 15, a difference thermal imagegeneration unit 16, a heat trace region extraction unit 17, and aninformation output unit 181. The actual object camera 210 and thethermal camera 22 are connected to the heat trace region extractiondevice 10 of the fourth embodiment, and the images taken by the actualobject camera 210 and the thermal camera 22 are input to the heat traceregion extraction device 10.

The heat trace region extraction method of the fourth embodiment isrealized by each unit of the heat trace region extraction deviceexecuting the processing of steps S101 to S109 shown in FIG. 21 andbelow. Hereinafter, the portions different from the first embodiment tothe third embodiment will be mainly described. Duplicate explanationswill be omitted for the same portions as those of the first to thirdembodiments.

The actual object camera 210, the actual object image acquisition unit110, the background actual object image generation unit 120, thedifference actual object image generation unit 130, the thermal camera22, the thermal image acquisition unit 14, the background thermal imagegeneration unit 15, the difference thermal image generation unit 16, andthe heat trace region extraction unit 17 are the same as the actualobject camera 210, the actual object image acquisition unit 110, thebackground actual object image generation unit 120, the differenceactual object image generation unit 130, the thermal camera 22, thethermal image acquisition unit 14, the background thermal imagegeneration unit 15, the difference thermal image generation unit 16, andthe heat trace region extraction unit 17 of the modified example of thefirst embodiment, respectively.

However, the actual object camera 210 is a camera in which thewavelength band of the electromagnetic waves to be photographedsatisfies the requirements described later. Further, it is assumed thatthe information indicating the heat trace region extracted by the heattrace region extraction unit 17 is input to the information output unit181.

In the fourth embodiment, the information output unit 181 has a functionof a projector. Similarly to the heat trace region output unit 18 of thefirst embodiment, the information output unit 181 having the projectorfunction projects information indicating the heat trace region withvisible light to a certain range which is a photographing range (S109).

In the fourth embodiment, the actual object camera 210 is a camera thatdoes not photograph the wavelength band of visible light projected bythe projector. That is, the actual object camera 210 is a camera thatphotographs electromagnetic waves in a wavelength band different fromthe wavelength band of visible light projected by the projector.Further, naturally, the actual object camera 210 is a camera that doesnot photograph the heat emitted by the actual object. That is, theactual object camera 210 is a camera that photographs electromagneticwaves in a wavelength band different from the wavelength band of theelectromagnetic waves obtained by the thermal camera 22. Further,naturally, since the information output unit 181 projects visible light,the information output unit 181 projects visible light in a wavelengthband different from the wavelength band of the electromagnetic wavesobtained by the thermal camera 22. From the above, the wavelength bandof the electromagnetic waves obtained by the actual object camera 210,the wavelength band of the electromagnetic waves obtained by the thermalcamera 22, and the wavelength band of the visible light projected by theinformation output unit 181 do not overlap. The wavelength band is arange of wavelengths in which the intensity of electromagnetic waves orvisible light is equal to or higher than a predetermined intensity.

For example, the wavelength band of visible light projected by theinformation output unit 181 is set to 400 nm to 780 nm, the wavelengthband of electromagnetic waves obtained by the actual object camera 210is set to the wavelength band in the near-infrared region (for example,780 nm to 2500 nm), and the wavelength band of the electromagnetic wavesobtained by the thermal camera 22 is set to 8 μm to 14 μm, which is thewavelength band of heat rays (far-infrared region). The wavelength bandin the near-infrared region may be 800 nm to 1000 nm. The upper limit ofthe wavelength band of visible light projected by the information outputunit 181 may be set to 800 nm, the lower limit of the wavelength band ofthe electromagnetic waves obtained by the actual object camera 210 maybe set to 850 nm, and the wavelength band of the electromagnetic wavesobtained by the thermal camera 22 may be set to 8 μm to 14 μm.

For example, the wavelength band of visible light projected by theinformation output unit 181 may be set to 400 nm to 600 nm, the lowerlimit of the wavelength band of electromagnetic waves obtained by theactual object camera 210 may be set to 600 nm, which is the wavelengthof visible light, and the wavelength band of the electromagnetic wavesobtained by the thermal camera 22 may be set to 8 μm to 14 μm. As inthis example, if the wavelength bands do not overlap, both thewavelength band of the electromagnetic waves obtained by the actualobject camera 210 and the wavelength band of the visible light projectedby the information output unit 181 may be the wavelength bands of thevisible light.

For example, the wavelength band of visible light projected by theinformation output unit 181 may be set to 600 nm to 800 nm, thewavelength band of the electromagnetic waves obtained by the actualobject camera 210 may be set to 400 nm to 600 nm, and the wavelengthband of the electromagnetic waves obtained by the thermal camera 22 maybe set to 8 μm to 14 μm. As in this example, as long as the wavelengthbands do not overlap, the order of the wavelength band of visible lightprojected by the information output unit 181 and the wavelength band ofthe electromagnetic waves obtained by the actual object camera 210 doesnot matter. That is, it is not essential that the wavelength band of theelectromagnetic waves obtained by the actual object camera 210 islocated on the longer wavelength side than the wavelength band of thevisible light projected by the information output unit 181. Thewavelength band of the visible light projected by the information outputunit 181 may be located on the longer wavelength side than thewavelength band of the electromagnetic waves obtained by the actualobject camera 210.

The wavelength band may be controlled by a well-known technique. Thewavelength band can be controlled by attaching a physical filter fortransmitting electromagnetic waves and visible light of a predeterminedwavelength band to a lens of the actual object camera 210, a lens of thethermal camera 22, and a projection lens of a projector of theinformation output unit 181. The wavelength band may be controlled bysignal processing. For example, the information output unit 181 maycontrol the wavelength band by filtering the signal generated forprojection and projecting an image based on the filtered signal. Theactual object camera 210 may control the wavelength band by filteringthe signal acquired by the actual object camera 210 and acquiring anactual object image from the filtered signal. Similarly, the thermalcamera 22 may control the wavelength band by filtering the signalacquired by the thermal camera 22 and acquiring a thermal image from thefiltered signal.

In the heat trace region extraction device and method of the modifiedexample of the first embodiment and the second and third embodiments,the wavelength band of the electromagnetic waves obtained by the actualobject camera 210, the wavelength band of the electromagnetic wavesobtained by the thermal camera 22, and the wavelength band of thevisible light projected by the information output unit 181 may beprevented from overlapping with each other in the same manner asdescribed above. That is, the fourth embodiment also includes anembodiment in which the heat trace region extraction device and methodof the modified example of the first embodiment and the second and thirdembodiments are configured such that the wavelength band of theelectromagnetic waves obtained by the actual object camera 210, thewavelength band of the electromagnetic waves obtained by the thermalcamera 22, and the wavelength band of the visible light projected by theinformation output unit 181 are prevented from overlapping with eachother in the same manner as described above.

In this way, since the wavelength band of the electromagnetic wavesobtained by the actual object camera 210, the wavelength band of theelectromagnetic waves obtained by the thermal camera 22, and thewavelength band of the visible light projected by the information outputunit 181 do not overlap with each other, the visible light projected bythe information output unit 181 is not photographed by the actual objectcamera 210. Therefore, the heat trace region extraction device andmethod of the fourth embodiment can appropriately extract the heat traceregion.

If the wavelength band of the electromagnetic waves obtained by theactual object camera 210 does not include the wavelength band of theelectromagnetic waves existing in the photographing range due toillumination or the like, the photographing range may be irradiated bythe irradiator 41 that emits the electromagnetic waves in the wavelengthband obtained by the actual object camera 210. For example, when thewavelength band of the electromagnetic waves obtained by the actualobject camera 210 is the wavelength band in the near-infrared region andthe illumination in the photographing range is an illumination such as afluorescent lamp that does not include the wavelength band in thenear-infrared region, the photographing range may be irradiated by theirradiator 41 that emits the electromagnetic waves in the wavelengthband in the near-infrared region obtained by the actual object camera210. In this case, the heat trace region extraction system 1 of thefourth embodiment may further include the irradiator 41, for example, asindicated by the broken line in FIGS. 7, 9, 13, and 20 .

Fifth Embodiment

In the heat trace region extraction device and method of the first tofourth embodiments, the information output unit 181 may outputinformation indicating the heat trace region in various variations. Theheat trace region extraction device and method of the fifth embodimentis the heat trace region extraction device and method of the first tofourth embodiments, in which the information output unit 181 outputsinformation indicating the heat trace region in various variations. Forthe sake of simplicity, in the fifth embodiment, the heat trace regionoutput unit 18 of the first embodiment and the modified example of thefirst embodiment will be referred to as an information output unit 181.

As described above, the information output unit 181 outputs informationindicating a heat trace region which is a region of a trace of heat,extracted based on an actual object image which is an image of an actualobject, obtained by photographing a certain range with an actual objectcamera for photographing the actual object and a thermal image which isan image of the heat emitted by the actual object, obtained byphotographing a certain range with a thermal camera for photographingthe heat emitted by the actual object. Hereinafter, the portionsdifferent from the heat trace region extraction device and method of thefirst to fourth embodiments will be mainly described. Duplicateexplanations will be omitted for the same portions as those in the firstto fourth embodiments.

(Variation 1)

The information output unit 181 may display information indicating theheat trace region on a transmissive display. An example of atransmissive display is a transmissive head-mounted display. In thiscase, the information output unit 181 includes a transmissive display,and detects the position and direction of the transmissive display sothat the heat trace region existing in the real space (that is, thephotographing range of the actual object camera and the thermal camera)visually recognized by the user through the transmissive display andinformation indicating the heat trace region displayed on thetransmissive display are aligned. As a result, the user can visuallyrecognize the information indicating the heat trace region displayed onthe transmissive display as if it is superimposed and displayed in thereal space that is visually recognized through the transmissive displayand immediately understand the place where the virus may be attached.

(Variation 2)

The information output unit 181 may display the information indicatingthe heat trace region on a display so as to be superimposed on an imageor a video in a certain range which is the photographing range.

Examples of the display include a digital signage, a smartphone, atablet terminal, an electric bulletin board, and a TV. For example, theinformation output unit 181 may display information indicating the heattrace region on a display so as to be superimposed on an actual objectimage or a visible image. The actual object image or visible image isthe actual object image or visible image acquired by the actual objectimage acquisition unit 110 or the visible image acquisition unit 11. Forexample, the information output unit 181 may display informationindicating the heat trace region on a display so as to be superimposedon an image or video in a certain range, which is a photographing range,taken by a camera different from the visible light camera 21 and theactual object camera 210. An example of a camera different from thevisible light camera 21 and the actual object camera 210 is a cameraprovided in a smartphone or a tablet terminal.

Note that, for example, using AR (Augmented Reality) technology,information indicating the heat trace region may be superimposed on animage or video in a certain range, which is the photographing range. Forexample, the information output unit 181 detects the position anddirection of the display of a smartphone or a tablet terminal to displayan image or a video in which the information indicating the heat traceregion is superimposed on an image of a video in a certain range on thedisplay so that the actual object existing in the real space (that is,the photographing range of the actual object camera and the thermalcamera) visually recognized by the user without the display and theimage of the actual object included in the image or the video displayedon the display are aligned. In this case, since the real space visuallyrecognized by the user without the display and the space displayed onthe display are aligned in real time according to the position and theangle of the smartphone or the tablet terminal, the user can moreimmediately understand the place where the virus or the like may beattached.

(Variation 3)

The information output unit 181 may output the information indicatingthe heat trace region via sound. For example, the information outputunit 181 may output a voice such as “X people touched here”. For thispurpose, the information output unit 181 may store the number of times xthe heat trace region was extracted. The information output unit 181 mayoutput a warning sound when the number of times the heat trace regionwas extracted after disinfection exceeds a predetermined number oftimes. The warning sound may be voice or non-voice.

The information output unit 181 may change the presence/absence of awarning and/or the warning sound according to the movement of the user.

For example, the information output unit 181 may output a soundexpressed in a stronger expression method as the distance between theheat trace region and a person or the hand of the person included in thevisible image or the actual object image photographed by the visiblelight camera 21 or the actual object camera 210 decreases. An example ofa sound expressed in a stronger expression method as the distancebetween the heat trace region and the person or the hand of the persondecreases is that the sound is louder as the distance between the heattrace region and the person or the hand of the person decreases. Theinformation output unit 181 may output information indicating the heattrace region via sound only when the distance between the heat traceregion and the person or the hand of the person included in the visibleimage or the actual object image photographed by the visible lightcamera 21 or the actual object camera 210 is equal to or less than apredetermined distance.

The information output unit 181 may output a sound expressed in astronger expression method as the distance between the heat trace regionand the person or the hand of the person decreases using two or moredirectional speakers having different sound transmission regions. Inthis case, the information output unit 181 performs control so that thedirectional speaker whose distance between the heat trace region and thesound transmission region is shorter outputs the sound expressed in astronger expression method.

(Variation 4)

The information output unit 181 may display the information indicatingthe heat trace region via text. For example, the information output unit181 may send a text corresponding to the information indicating the heattrace region using an e-mail, a short message service, a notification ormessage function in an Social Network Service (SNS). The textcorresponding to the information indicating the heat trace region istransmitted to, for example, a manager of the photographing range and aperson who disinfects the photographing range. The destination of thetext corresponding to the information indicating the heat trace regionmay be one or more destinations, or may be two or more destinations.

The information output unit 181 may store the number of times x the heattrace region was extracted, and output a text corresponding to theinformation indicating the heat trace region when the number of timesthe heat trace region was extracted exceeds a predetermined number oftimes. The information output unit 181 may project a text such as“someone touched the door” with a projector or display it on a display.Examples of the display here include a digital signage, an electricbulletin board, and a TV installed near the photographing range.

(Variation 5)

The information output unit 181 may output information indicating aregion other than the heat trace region instead of the informationindicating the heat trace region. The region other than the heat traceregion is a region where the temperature has not risen due to contactwith a person, that is, a region where it is unlikely that a virus suchas a new coronavirus is attached. By showing a region where it isunlikely that a virus such as a new coronavirus is attached, it ispossible to present the user with a region that is safe to touch.

For example, the information output unit 181 may brightly displayinformation indicating a region other than the heat trace region. Theinformation output unit 181 may divide a region other than the heattrace region into a plurality of partial regions, randomly select one ofthe partial regions, and brightly display information indicating theselected partial region. As a result, the information output unit 181can present the user with a region that is safe to touch, for example,in the handrail. The information output unit 181 may display informationindicating a region other than the heat trace region brighter than theinformation indicating the heat trace region. For this purpose, forexample, the information output unit 181 displays the photographingrange brightly in advance, and when the heat trace region is extracted,the extracted heat trace region is darkly displayed.

(Variation 6)

The information output unit 181 may display information indicating theheat trace region in a different expression method depending on thenumber of times the heat trace region was extracted. That is, theinformation output unit 181 may output different information asinformation indicating the heat trace region, depending on the number oftimes the heat trace region was extracted. For example, the informationoutput unit 181 may display a region in a stronger expression method asthe number of times the region was extracted as the heat trace regionincreases. For example, as illustrated in FIG. 22 , the informationoutput unit 181 may display a region in a darker color as the number oftimes the region was extracted as the heat trace region increases. Bydoing so, it is possible to inform the user that the darker the region,the more it should not be touched or should be disinfected.

Although an embodiment of the present invention has been described indetail above, the present invention is not limited to the specificembodiment described above, and various modified example and changes canbe made within the concept of the present invention described in theclaims.

1. A device for extracting a heat trace region, the device comprising aprocessor configured to execute a method comprising: outputtinginformation indicating a heat trace region which is a region of a traceof heat, wherein the region of a trace of heat is extracted based on anactual object image and a thermal image, the actual object imageincludes an image of an actual object, the actual object image is basedon photographing a predetermined range with an actual object camera forphotographing the actual object and a thermal image, the thermal imageincludes an image of the heat emitted by the actual object, and thethermal image is based on photographing the predetermined range with athermal camera for photographing the heat emitted by the actual object;and displaying information, wherein the information indicates the heattrace region on a transmissive display, and the heat trace region in thepredetermined range is in alignment with the information indicating theheat trace region displayed on the transmissive display with each other.2. The device according to claim 1, wherein wherein the informationincludes video data.
 3. The device according to claim 1, wherein theinformation includes sound data.
 4. The device according to claim 1,wherein the information includes text data.
 5. The device according toclaim 1, wherein the information indicates an another region that isother than the region of a trace of heat, extracted based on the actualobject image and the thermal image.
 6. The device according to claim 1,wherein the information is based on on the number of times the heattrace region was extracted.
 7. The device according to claim 1, theprocessor further configured to execute a method comprising: generatinga difference actual object image, wherein the difference actual objectimage includes an image of a difference between the actual object imageand a background actual object image, and the background actual objectimage corresponds to an a background image of the predetermined range;generating a difference thermal image, wherein the difference thermalimage includes an image of a difference between the thermal image and abackground thermal image, and the background thermal image correspondsto a thermal image of the background; and extracting the heat traceregion by removing a region of the actual object from the thermal imagebased on the difference actual object image and the difference thermalimage.
 8. A method for extracting a heat trace region, comprising:outputting information indicating a heat trace region which is a regionof a trace of heat, wherein the region of a trace of heat is extractedbased on an actual object image and a thermal image, the actual objectimage includes an image of an actual object, the actual object image isbased on photographing a predetermined range with an actual objectcamera for photographing the actual object and a thermal image, thethermal image includes an image of the heat emitted by the actualobject, and the thermal image is based on photographing thepredetermined range with a thermal camera for photographing the heatemitted by the actual object; and displaying information, wherein theinformation indicates the heat trace region on a transmissive display,and the heat trace region in the predetermined range is in alignmentwith the information indicating the heat trace region displayed on thetransmissive display with each other.
 9. The method according to claim8, wherein the information includes video data.
 10. The method accordingto claim 8, wherein wherein the information includes sound data.
 11. Themethod according to claim 8, wherein the information includes text data.12. The method according to claim 8, wherein the information indicatesan another region that is other than the region of a trace of heat,extracted based on the actual object image and the thermal image. 13.The method according to claim 8, wherein the information is based on thenumber of times the heat trace region was extracted as informationindicating the heat trace region.
 14. A computer-readable non-transitoryrecording medium storing computer-executable program instructions thatwhen executed by a processor cause a computer system to execute a methodcomprising: outputting information indicating a heat trace region whichis a region of a trace of heat, wherein the region of a trace of heat isextracted based on an actual object image and a thermal image, theactual object image includes an image of an actual object, the actualobject image is based on photographing a predetermined range with anactual object camera for photographing the actual object and a thermalimage, the thermal image includes an image of the heat emitted by theactual object, and the thermal image is based on photographing thepredetermined range with a thermal camera for photographing the heatemitted by the actual object; and displaying information, wherein theinformation indicates the heat trace region on a transmissive display,and the heat trace region in the predetermined range is in alignmentwith the information indicating the heat trace region displayed on thetransmissive display with each other.
 15. The computer-executablenon-transitory recording medium according to claim 14, wherein theinformation includes video data.
 16. The computer-executablenon-transitory recording medium according to claim 14, wherein theinformation includes sound data.
 17. The computer-executablenon-transitory recording medium according to claim 14, wherein theinformation includes text data.
 18. The computer-executablenon-transitory recording medium according to claim 14, wherein theinformation indicates an another region that is other than the region ofa trace of heat, extracted based on the actual object image and thethermal image.
 19. The computer-executable non-transitory recordingmedium according to claim 14, wherein the information is based on thenumber of times the heat trace region was extracted as informationindicating the heat trace region.
 20. The computer-executablenon-transitory recording medium according to claim 14, thecomputer-executable program instructions when executed further causingthe computer system to execute a method comprising: generating adifference actual object image, wherein the difference actual objectimage includes an image of a difference between the actual object imageand a background actual object image, and the background actual objectimage corresponds to a background image of the predetermined range;generating a difference thermal image, wherein the difference thermalimage includes an image of a difference between the thermal image and abackground thermal image, and the background thermal image correspondsto a thermal image of the background; and extracting the heat traceregion by removing a region of the actual object from the thermal imagebased on the difference actual object image and the difference thermalimage.